KOSAC Lexicon — User Manual¶

kosac-lexicon packages a morpheme-level Korean sentiment lexicon derived from KOSAC (the Korean Sentiment Analysis Corpus). The distribution name is kosac-lexicon; the import name is kosac.

In a hurry? Jump to §4 Quick start. For the recommended end-to-end API see §6 SentimentAnalyzer. (한국어 매뉴얼: manual.ko.md.)

1. Introduction¶

The lexicon was built on the assumption that a word’s sentiment derives from the core subjective expressions (“Seeds”) that contain it: morpheme N-grams extracted from KOSAC’s Seeds are the lexicon entries. The data was released in 2016 and is fixed (data version 2016).

A single text flows through the pipeline like this:

Korean sentence → (POS tagging) surface/POS tokens → (lexicon matching)
sentiment expressions → label probability distribution

2. Installation¶

pip install kosac-lexicon                  # core: lexicon data + query API (pandas/numpy only)
pip install "kosac-lexicon[kiwi]"          # + Kiwi POS tokenizer (pure pip, no Java)
pip install "kosac-lexicon[transformers]"  # + HuggingFace subword tokenizer
pip install "kosac-lexicon[sklearn]"       # + scikit-learn feature extractor
pip install "kosac-lexicon[all]"           # everything

No Java (JVM) is required. POS tagging uses Kiwi (kiwipiepy), which installs from PyPI.
The core install alone loads, queries, and filters the lexicon data. Add [kiwi] only when you need to tokenize raw sentences.

3. Core concepts¶

3.1 Entry format¶

An entry is an N-gram of surface/POS tokens joined by spaces (Sejong tagset):

unigram: 좋/VA, 힘/NNG
bigram: 행복/NNG 하/XSA

Each lexicon has 16,361 entries (3,476 unigrams · 6,579 bigrams · 6,307 trigrams).

3.2 The six semantic features¶

List them with kosac.FEATURES.

Feature	Label values
`polarity`	COMP, NEG, NEUT, None, POS
`intensity`	High, Low, Medium, None
`expressive-type`	dir-action, dir-explicit, dir-speech, indirect, writing-device
`nested-order`	0, 1, 2, 3
`subjectivity-polarity`	COMP, NEG, NEUT, POS
`subjectivity-type`	Agreement, Argument, Emotion, Intention, Judgment, Others, Speculation

kosac.describe_feature("polarity") summarises each label value (§10).

In addition to the six canonical features, v0.5.0 ships a derived polarity-blend lexicon (POS/NEG) tuned for classification accuracy — loadable by name but deliberately kept out of kosac.FEATURES. See §6.5.

3.3 Lexicon data structure¶

A lexicon is a pandas DataFrame indexed by entry (space-joined morphemes), with:

(per-label columns): the absolute count of Seeds with each label value — the only data stored in the CSV
freq: number of Seeds (= the row total), derived by the loader
max.value / max.prop: the most frequent label and its proportion, derived by the loader from the counts

4. Quick start¶

import kosac

# (1) Query the lexicon only — no tokenizer / Java needed
lex = kosac.load_lexicon("polarity", ngrams=[1], min_freq=5)
lex.get_entry("힘/NNG")

# (2) Analyze a sentence — needs Kiwi: pip install "kosac-lexicon[kiwi]"
from kosac import SentimentAnalyzer

analyzer = SentimentAnalyzer("polarity")
result = analyzer.analyze("이 영화 정말 좋다")
result["features"]["polarity"]["label"]   # 'POS'

5. Working with lexicons (low-level API)¶

5.1 Loading¶

import kosac
from kosac.lexicon import PolarityLexicon

lex = kosac.load_lexicon("polarity", ngrams=[1, 2, 3], min_freq=0, threshold=0.0)
lex = PolarityLexicon.load(ngrams=[1])                 # equivalent classmethod
lex = PolarityLexicon(filepath="my-lexicon.csv")       # your own CSV

ngrams: which N-gram lengths to keep (default [1]).
feature in load_lexicon must be one of kosac.FEATURES.

5.2 Filtering & querying¶

lex.set_lexicon(min_freq=5, threshold=0.6)  # keep freq>=5 AND max.prop>0.6
lex.set_lexicon(min_freq=1)                 # loosen again — re-filters the original
lex.get_size()              # number of entries
lex.get_labels()            # ['COMP','NEG','NEUT','None','POS']
lex.get_lexicon()           # current (filtered) DataFrame
lex.get_original_lexicon()  # the unfiltered original
lex.get_entry("좋/VA")      # one entry's row  (legacy alias: lex.get("좋/VA"))

set_lexicon always filters the original lexicon, so a threshold can be tightened or loosened again.

5.3 Matching & scoring (needs a tokenizer)¶

from kosac.tokenizers import KiwiTokenizer
tok = KiwiTokenizer()

lex.match_patterns("나는 정말 행복하다", tok)   # matched entry strings (alias: lex.match)
lex.get_match_info("나는 정말 행복하다", tok)   # (entry, max.value, max.prop) tuples
lex.get_sent_probs("나는 정말 행복하다", tok)   # per-label probabilities (softmax) Series

sorting=True (default) matches longer N-grams / higher-probability entries first; sorting=False matches in token order.

For richer results (character spans, negation handling, …) use the analyzer; match_patterns / get_sent_probs are plain lexicon lookups.

5.4 Building & extending a custom lexicon¶

from kosac.lexicon import GenericLexicon
from kosac.corpora import Corpus

lex = GenericLexicon(ngrams=[1, 2])
lex.set_labels(["POS", "NEG"])           # GenericLexicon only
lex.add_token("좋/VA", "POS")            # +1 count for a label on an entry
lex.update([("싫/VA", "NEG"), ("좋/VA", "POS")])

corpus = Corpus("data/example.csv")      # headerless text,label CSV
lex.update_from_corpus(corpus, KiwiTokenizer())

lex.export_user_dict("user_dictionary.txt")  # export unigrams as form\tPOS

lex.save("my-lexicon.csv")               # save in the package CSV format (ngram + counts)
GenericLexicon(filepath="my-lexicon.csv", ngrams=[1])   # reload (labels inferred)

6. SentimentAnalyzer (high-level API)¶

SentimentAnalyzer bundles lexicons with a tokenizer and scores text in one call. This is the recommended entry point.

from kosac import SentimentAnalyzer

analyzer = SentimentAnalyzer(
    features="polarity",     # also 'all', 'polarity-blend', or ['polarity', ...]
    tokenizer=None,          # default KiwiTokenizer()
    ngrams=(1, 2, 3),
    min_freq=0, threshold=0.0,
    smoothing=True,
    scoring="multiscale",    # default; "greedy" for the legacy matcher (§6.5)
    ngram_weights=None,      # optional per-length weights, e.g. {1:1, 2:1, 3:1}
    align=False,             # seed Kiwi's user dict from the lexicon (§7.3)
    negation=False,          # negation handling (§6.3)
    intensifier=False,       # intensifier handling (§6.3)
    window=2, intensifier_factor=2.0,
)

Since v0.5.0 the default scorer is multi-scale (§6.5): it sums every overlapping n-gram match. For unigram-only use (ngrams=[1]) it is identical to the legacy scoring="greedy"; it differs only when bigrams/trigrams match.

6.1 The `analyze(text)` result¶

r = analyzer.analyze("이 영화 정말 좋다")

{
  "text": "이 영화 정말 좋다",
  "tokens": ["이/MM", "영화/NNG", "정말/MAG", "좋/VA", "다/EF"],
  "features": {
    "polarity": {
      "label": "POS",                # top label
      "prob": 0.97,                  # its probability
      "probs": {"COMP": ..., "NEG": ..., "POS": 0.97, ...},
      "matches": [
        {"entry": "좋/VA", "span": [8, 9], "text": "좋",
         "max_value": "POS", "max_prop": 0.92,
         "negated": False, "weight": 1.0},
        ...
      ]
    }
  }
}

span is a character offset [start, end) into the original text, so text[span[0]:span[1]] recovers the surface string.
The result is a plain, JSON-serialisable dict.

6.2 Multiple features at once / batch¶

SentimentAnalyzer("all").analyze("이 영화는 정말 좋았고 너무 행복했다")
# polarity=POS, intensity=Medium, expressive-type=dir-speech,
# nested-order=1, subjectivity-polarity=POS, subjectivity-type=Argument

analyzer.analyze_batch(["좋다", "싫다"])     # list of result dicts
analyzer.analyze_frame(["좋다", "싫다"])     # tidy pandas DataFrame
#    text  polarity.label  polarity.prob
# 0   좋다             POS            ...

6.3 Negation & intensifier¶

Opt-in heuristics. If a marker appears within window tokens of a matched expression, it is applied.

a = SentimentAnalyzer("polarity", negation=True, intensifier=True)
a.analyze("이 영화는 안 좋다")["features"]["polarity"]["label"]   # 'NEG'

Negation (안/못/않/없 …): swaps the POS↔NEG mass of the matched expression. Applied only to features whose labels contain both POS and NEG (polarity, subjectivity-polarity).
Intensifier (정말/너무/매우 …): multiplies the matched expression’s weight by intensifier_factor.
Marker sets are kosac.analyzer.DEFAULT_NEGATIONS / DEFAULT_INTENSIFIERS and can be replaced via the negations= / intensifiers= constructor arguments.

from kosac.analyzer import DEFAULT_INTENSIFIERS
SentimentAnalyzer("polarity", intensifier=True,
                  intensifiers=DEFAULT_INTENSIFIERS | {"완전/MAG"}, window=3)

⚠️ This is a window-based heuristic, so an adjacent morpheme may occasionally be flagged as negated. The final label is usually unaffected, but tune window or post-process if you need precise negation scope.

6.4 Frequency-count method¶

You can classify text by counting sentiment words (numbers and proportions of positive/negative words) instead of using probabilities — the approach common in content analysis and social-science studies. count() provides this: it tallies matched morphemes by their dominant label (max.value).

a = SentimentAnalyzer("polarity")
c = a.count("빗물이 흐르고 내 눈물도 흐르고 잃어버린 첫사랑도 흐르네")["features"]["polarity"]
c["counts"]       # {'NEG': 6, 'POS': 3, ...} — word counts per label
c["proportions"]  # per-label share (sums to 1)
c["total"]        # number of matched words
c["label"]        # most frequent label → 'NEG'

a.count_batch(texts)   # list of result dicts
a.count_frame(texts)   # per-document label counts (<feature>.POS, <feature>.NEG, ...)

analyze() (probabilities) and count() (word counts) aggregate the same matches differently. Negation/intensifier do not apply to the count method.

6.5 High-accuracy polarity (multi-scale + `polarity-blend`)¶

For POS/NEG classification, two v0.5.0 additions make the analyzer much stronger, especially on out-of-domain text:

Multi-scale scoring (the default). The legacy matcher was greedy leftmost-longest and non-overlapping, so a matched trigram suppressed its unigrams. Multi-scale instead sums every overlapping n-gram match, which strictly improves held-out sentiment classification. Use scoring="greedy" for the old behavior, or ngram_weights to weight the scales:

SentimentAnalyzer("polarity", scoring="greedy")            # legacy matcher
SentimentAnalyzer("polarity", ngram_weights={1: 1, 2: 1, 3: 1})  # equal (default)

count() always uses the non-overlapping matcher (correct for word tallies); multi-scale applies to analyze() / polarity_score().

The polarity-blend lexicon. The frozen polarity lexicon is near-chance out of domain. polarity-blend is a derived POS/NEG lexicon — the frozen KOSAC seeds blended with a lexicon learned from the NSMC corpus — that transfers far better (NIKL out-of-domain balanced-accuracy ≈ 0.53 → 0.73). It ships gzipped and is loadable by name, but is not one of the six canonical kosac.FEATURES, so SentimentAnalyzer("all") is unchanged.

clf = SentimentAnalyzer("polarity-blend")     # multi-scale scoring by default
clf.predict_polarity("이 영화 정말 재미있다")    # 'POS'
clf.polarity_score("시간 낭비 최악의 영화")      # negative float: P(POS) − P(NEG)
clf.predict_polarity_batch(reviews)           # list of 'POS'/'NEG'

polarity_score(text) → continuous P(POS) − P(NEG) in [-1, 1].
predict_polarity(text, threshold=0.0) → 'POS' if the score exceeds the threshold (raise it to trade recall for precision on imbalanced data).

The blend is derived data (CC BY-SA; see §12). Regenerate it — or build the larger “champion” variant — with python -m benchmarks.build_shipped_blend [--full]; the full benchmark methodology and numbers live in benchmarks/README.md.

7. Tokenizers¶

Module kosac.tokenizers.

7.1 Kinds¶

Class	Description	Extra
`Tokenizer`	whitespace split (base); for pre-tagged test input	none
`KiwiTokenizer`	Kiwi morpheme analysis (recommended); emits `surface/POS`	`[kiwi]`
`HuggingFaceTokenizer`	HuggingFace subwords	`[transformers]`

7.2 Common methods¶

tok.tokenize("나는 좋다")               # ['나/NP', '는/JX', '좋/VA', '다/EF']
tok.get_tokens_str("나는 좋다")         # '나/NP 는/JX 좋/VA 다/EF'
tok.tokenize_with_offsets("나는 좋다")  # [('나/NP', 0, 1), ('는/JX', 1, 2), ...]
tok.get_ngrams("나는 좋다", ns=[1, 2]) # unigram + bigram strings

7.3 Kiwi user-dictionary alignment¶

Registering lexicon entries as Kiwi user words makes its segmentation align better with the lexicon (easing the tagset mismatch).

from kosac.tokenizers import KiwiTokenizer

tok = KiwiTokenizer(user_words=[("행복하", "VA")])   # register directly
tok.add_user_words(["좋/VA", ("멋지", "VA")])         # 'form/tag' or (form, tag)

lex = kosac.load_lexicon("polarity", ngrams=[1])
tok = KiwiTokenizer.from_lexicon(lex, tags={"NNG", "VV", "VA", "XR"})  # seed from lexicon unigrams

SentimentAnalyzer(..., align=True) applies from_lexicon automatically.

8. Command-line interface¶

Installing adds a kosac command (or use python -m kosac).

kosac analyze "이 영화 정말 좋다"                 # pretty JSON (probabilities)
kosac analyze "이 영화는 안 좋다" --negation       # negation handling
kosac analyze "좋다" --features all                # all six features
kosac analyze "빗물이 흐르고 눈물도 흐르고" --count  # frequency-count method
kosac features                                     # list features
kosac citation                                     # BibTeX

printf '좋다\n싫다\n' | kosac analyze --compact     # one JSON object per line (JSONL)

analyze options: --features (comma-separated or all), --ngrams, --min-freq, --negation, --intensifier, --align, --count, --compact. With no text argument it reads one sentence per line from stdin.

9. scikit-learn integration¶

KosacVectorizer turns text into <feature>=<label> probability features (needs the [sklearn] extra).

from kosac.sklearn import KosacVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

clf = make_pipeline(KosacVectorizer("all", negation=True), LogisticRegression())
clf.fit(train_texts, labels)
clf.predict(test_texts)

vec = KosacVectorizer("polarity").fit(train_texts)
vec.get_feature_names_out()   # ['polarity=COMP', 'polarity=NEG', ...]

Constructor arguments mirror SentimentAnalyzer (features, ngrams, min_freq, negation, intensifier, align, tokenizer).

10. Utilities¶

import kosac

kosac.FEATURES            # tuple of the six feature names
kosac.__version__         # package version (e.g. '1.0.0')
kosac.__data_version__    # data vintage ('2016')

kosac.citation()                    # BibTeX string
kosac.describe_feature("polarity")  # {'feature','values','reference'}
kosac.FEATURE_VALUES["intensity"]   # label value -> description

11. FAQ & caveats¶

Do I need Java? No. Kiwi installs from PyPI.
Tagset differences. Entries use the Sejong tagset. Kiwi’s tagset is also Sejong-based, so the common morpheme tags (NNG/VV/VA/JKS/EC …) match, but some symbol/web tags and segmentations may differ from the tagger originally used to build KOSAC. Use align=True to mitigate.
Negation is not perfect. See the heuristic caveat in §6.3.
Wildcard entries. Some entries contain the regex-special * (e.g. 가*/JKS); they are escaped internally and matched literally.
No matches. If nothing matches, a feature’s label is None and probs is an empty dict (count returns total == 0).

12. Citation & license¶

print(kosac.citation())

Code: MIT (LICENSE)
Lexicon data (kosac/data/*.csv): CC BY-SA 4.0, derived from KOSAC (Seoul National University) (src/kosac/data/LICENSE). When redistributing, keep the attribution and share-alike terms.
polarity-blend (kosac/data/polarity-blend.csv.gz): derived data, CC BY-SA 4.0 — it blends the CC BY-SA KOSAC seeds with the CC0 NSMC corpus, so the result is CC BY-SA. See src/kosac/data/polarity-blend.NOTICE.