Named Entity Recognition (NER) is one of the most structuring tasks in text annotation for natural language processing. Identifying and delimiting in a raw document the mentions of people, organizations, locations, dates, or domain-specific concepts gives language models the fine-grained understanding they need to extract structured information from unstructured text. NER is a prerequisite for a wide range of applications: information extraction from contracts, semantic indexing of medical records, knowledge graph population, internal search engine improvement, and large-scale brand monitoring in text streams.
Performing this work correctly, however, requires rigorous methodology. Unlike document classification tasks (where the annotator’s judgment covers a global meaning), text annotation for NER operates at the token level. Every boundary decision, every entity type choice directly affects the quality of the delivered corpus and, ultimately, the performance of the trained model. This extreme granularity turns every schema ambiguity into a reproducible error across thousands of documents the moment a project scales beyond a single annotator.
This article, a companion to our complete guide to NLP text annotation tasks, examines the three methodological dimensions that determine whether a NER project succeeds: entity schema design, boundary management, and the handling of nested or discontinuous entities.
Named Entity Recognition: the core task of text annotation
NER is a text annotation task that consists of detecting, in a token stream, named units belonging to predefined categories. These units represent real-world entities: a natural or legal person, a geographical location, a date, an amount, a product, or a regulatory concept. Identifying them in text is the foundation of every structured information extraction pipeline.
The appeal of NER lies in its cross-sector versatility. In healthcare, NER text annotation extracts drug names, diagnoses, and coded procedures from clinical notes. In law, it identifies contracting parties, legal references, and jurisdictions. In finance, it detects company names, stock prices, and market indices. In human resources, it pulls job titles, skills, and academic qualifications from thousands of resumes. In each case, the annotated corpus feeds an extractor capable of processing document volumes that no human team could analyze manually at a reasonable cost.
NER stands apart from other text annotation tasks by the precision it demands at the token level. Binary document classification, for instance, tolerates some uncertainty: the unit of work is the entire document, and the annotator’s judgment covers a global sense. NER, by contrast, requires a decision at every position: Is this mention an entity? What type? Does it start here or one word earlier? Does it end before or after the punctuation mark? These questions, posed hundreds of times per document and tens of thousands of times across a full project, turn every schema gap into a large-scale inconsistency.
Historically, NER is among the first tasks to have been benchmarked through shared community evaluations, including the well-known CoNLL campaigns from the early 2000s. The founding entity types (PERSON, ORGANIZATION, LOCATION, MISCELLANEOUS) still form the core of most general-purpose schemas today. Modern specialized applications have vastly expanded this perimeter, with schemas sometimes exceeding thirty entity types, which amplifies the training, consistency, and quality control challenges inherent in any text annotation project of this scope.
Named Entity Recognition is a text annotation task that consists of delimiting and typing reference mentions in a raw corpus. Its application spans sectors as diverse as healthcare, law, finance, and industry, and it is an indispensable prerequisite for any large-scale information extraction pipeline.
Designing the entity schema: the founding decision in text annotation
Before annotating the first document, the project team must define the entity schema. This is the blueprint of the NER text annotation project: it lists the entity types to be recognized, defines them precisely, and sets the rules for resolving ambiguous cases. A poorly designed schema compounds into massive inconsistencies as the corpus grows, and correcting a corpus of tens of thousands of documents after the fact is disproportionately more costly than investing in rigorous upfront design.
Selecting entity types
Generic NER schemas share a common core: PERSON, ORGANIZATION, LOCATION, DATE, TIME, AMOUNT, and PERCENTAGE. These seven types are sufficient for many monitoring or general-purpose indexing applications. Specialized schemas add domain-specific types: DRUG, DIAGNOSIS, and MEDICAL_PROCEDURE for healthcare; PARTY, CLAUSE, and JURISDICTION for legal; LISTED_COMPANY, INDEX, and FINANCIAL_INSTRUMENT for finance.
The temptation is to multiply types to cover every scenario upfront. This is a recurring mistake in NER text annotation: an overly granular schema generates inter-annotator inconsistencies, because the conceptual boundaries between closely related types become blurred: annotators hesitate between SYMPTOM and DIAGNOSIS, between EMPLOYER and ORGANIZATION. A better approach is to start with seven to twelve clearly defined types and extend the schema iteratively based on difficulties observed in production, rather than attempting to anticipate everything from the outset.
Writing entity definitions
Each entity type must be accompanied by an operational definition: what it includes, what it excludes, and how it differs from neighboring types. These definitions are supplemented by positive examples (mentions that should be annotated with this type) and contrastive examples (mentions that resemble the type but do not qualify, with the reason why).
In a medical schema, DRUG designates the names of molecules and branded drugs, but not generic therapeutic classes such as “antibiotic” or “anti-inflammatory,” which fall under a distinct THERAPEUTIC_CLASS type. The schema note specifies that “aspirin” is a DRUG, that “NSAID” is a THERAPEUTIC_CLASS, and that “the treatment” without an explicit designation receives no annotation. This documentation work, often rushed at project startup, directly determines the reproducibility of text annotation throughout the project and significantly reduces onboarding time for new annotators.
Validating the schema on a pilot batch
Before launching production, it is essential to annotate a pilot batch of one hundred to two hundred documents using the provisional schema. This batch surfaces cases the schema did not anticipate, types that annotators systematically confuse, and definitions that lack precision. After the pilot, the schema is revised, training is adjusted, and a baseline inter-annotator agreement is measured. Only at this point can full-scale production begin without a major risk of corpus corruption. Teams that skip this step to save time almost invariably find themselves re-annotating a substantial portion of the corpus a few weeks into production.
Entity boundary management: the central challenge of NER text annotation
Entity boundaries represent the most frequent source of friction in text annotation for NER. A boundary error produces one extra or one missing token, which degrades the model’s evaluation metrics and generates incorrect extractions in production conditions. Inter-annotator agreement studies consistently show that boundary disagreements are more frequent than type disagreements: annotators typically agree on what an entity is before they agree on exactly where it starts and ends.
Entity boundary management is the central challenge of NER text annotation. It precisely determines where each entity begins and ends in the text stream, and it directly conditions the quality and exploitability of the delivered corpus.
The maximal inclusion rule
Most NER annotation guides recommend applying the maximal inclusion rule: the entity span includes all tokens that semantically belong to it, including determiners, qualifying adjectives, and appositions that are part of the proper name or designation.
For example, “the United Nations” is annotated as a single ORGANIZATION entity covering three tokens, not reduced to “United Nations” without the determiner. Similarly, “the city of Berlin” is annotated across four words if the context refers to the municipal authority rather than a simple toponym occurrence. This rule simplifies annotators’ work by eliminating hesitations over determiners, but it must be applied consistently across the entire corpus without exceptions: even a single violation creates false negatives that the model learns and reproduces.
Determiners, prepositions, and punctuation
The most ambiguous boundary cases involve intercalated punctuation, prepositions, and parentheses. Should the comma following a PERSON entity in a list be included? No, the comma does not semantically belong to the entity. Should a parenthetical acronym be included if it refers to the same entity as the preceding phrase (e.g., “European Central Bank (ECB)”)? The annotation guide must rule explicitly: either annotate the acronym separately as a distinct entity, or incorporate the parenthetical into the main entity span.
Every punctuation rule must be spelled out in the guide with a real textual example drawn from the target corpus whenever possible. The absence of explicit rules generates massive inconsistencies, because each annotator makes different micro-decisions across the thousands of choices they face per working session. On a corpus of fifty thousand documents, a 2% inconsistency rate on parenthetical handling alone represents a thousand documents requiring correction.
Entities spanning syntactic boundaries
Some entities traverse unusual syntactic structures. An organization name can appear fragmented by a parenthetical remark, a document title can be split across two sentences in an automatic summary, or an entity can include an abbreviation followed by its expansion in parentheses. These cases must be catalogued and documented with real corpus examples during the pilot phase, because their handling cannot be reliably inferred from general rules without the risk of error.
Nested and discontinuous entities: the complex cases in text annotation
Text annotation for NER reaches its maximum complexity when entities are not linear and disjoint segments. Two situations present specific challenges: nested entities and discontinuous entities. Both require explicit methodological choices before production launches, as they have direct implications for tooling, annotator training, and the architecture of the trained model.
Nested and discontinuous entities represent the most demanding cases in text annotation for NER. Their handling requires adapted tagging schemes and explicit methodological choices from the project design phase onward.
Nested entities
A nested entity is an entity contained within another entity of a different type. In the phrase “the CEO of Apple Inc.”, the mention “Apple Inc.” is an ORGANIZATION entity, and the full phrase “the CEO of Apple Inc.” may be a ROLE_AND_AFFILIATION entity according to the schema. The two annotations partially overlap on the tokens “Apple Inc.”
The standard BIO format (Beginning, Inside, Outside) does not handle nesting: a position in the sequence can only receive a single tag. To handle nested entities, teams must use multi-layer formats, span-based annotation (character intervals rather than token sequences), or decide to simplify the schema by favoring a single annotation layer. This decision must be made before launch, as a mid-project format conversion is costly and risks information loss.
Tagging schemes
The tagging scheme is the formal representation of entities in the project’s text annotation output files. The BIO format (B for Beginning, I for Inside, O for Outside) is the most widespread: each token receives a tag indicating whether it opens an entity (B-TYPE), falls inside an entity (I-TYPE), or is outside any entity (O). The BIOES format adds two tags: S (Single) for a single-token entity, and E (End) for the last token of a multi-token entity. This extended format is often preferred for sequence models because it provides explicit signals on both boundaries of each entity.
The choice of tagging scheme must be aligned between the text annotation team and the machine learning engineers training the model. A format mismatch between the delivered corpus and the training pipeline generates costly conversions and silent errors that are difficult to diagnose during initial evaluation runs.
Discontinuous entities
A discontinuous entity is one whose constituent tokens are not adjacent in the text. They appear mainly in complex legal documents, technically structured documents with tabular content, or transcribed spoken corpus. For example, in “nitrogen and sulfur were both detected,” each chemical element forms a distinct entity that is not part of a contiguous span. These cases are rare in most standard NER schemas, but their handling must be defined explicitly in the annotation guide to prevent inter-annotator divergences from accumulating across the corpus.
Training and calibrating annotators for NER text annotation
Annotator training is decisive in any NER text annotation project. Systematic errors stemming from insufficient training propagate across tens of thousands of documents before being detected by quality control. Investing in initial training is always more cost-effective than correcting a corrupted corpus mid-production, and teams that allocate time to ongoing calibration observe measurable consistency gains across the full lifecycle of the project.
Training on disputed cases
Initial training should not be limited to schema presentation. It must expose annotators to real disputed cases, drawn from the target corpus, for which the correct answer is collectively debated and argued. This deliberation-based training format improves intra-annotator consistency from the first weeks of production and also serves as a valuable opportunity to identify gaps in the annotation guide before they propagate at scale. Refresher sessions are recommended whenever quality control reveals a drift, particularly on long-term projects involving team rotations.
Inter-annotator agreement
Inter-annotator agreement (IAA) measures the consistency level between annotators working on the same document. In NER text annotation, Cohen’s kappa is the most widely used metric, applied separately to entity types and to entity boundaries. An agreement below 0.70 on entity types indicates that the schema is ambiguous or that training is insufficient. An agreement below 0.60 on boundaries reveals divergences in the application of delimitation rules. In both cases, the corrective path involves revising the annotation guide and holding a recalibration session before resuming full-scale production.
Gold standards and quality control
Quality control in NER text annotation relies on two complementary tools. A gold standard is a set of documents annotated by a reference expert whose correct answers are known and fixed. It allows the objective measurement of each annotator’s precision and pinpoints the entity types where additional training is needed. Honeypots are gold-standard documents inserted unannounced into production batches to detect drifts in real time without waiting for formal review cycles. On projects with high quality requirements, 100% verification by a senior annotator-reviewer is the norm, supplemented by sampling by an independent quality department to ensure overall corpus consistency over the project’s duration.
Text annotation for NER in specialized domains
Text annotation needs for NER vary significantly across application domains. Each sector brings its own vocabularies, terminological ambiguities, and quality and compliance constraints. The versatility of a generic schema reaches its limits quickly when target entities are technical concepts that only a domain expert can delimit with precision.
Clinical NER
In healthcare data, text annotation for NER applies to highly technical documents: hospital discharge summaries, operative notes, biopsy reports, and consultation transcripts. Entities to annotate include drug names, diagnoses coded under ICD-10 or SNOMED CT, medical procedures, anatomical structures and organs, and laboratory results. A term like “NSAID” (non-steroidal anti-inflammatory drug) may not be recognized as a DRUG entity by a non-medical annotator, and clinical abbreviations are a near-systematic source of error without domain-specific training. Expert clinical oversight is recommended for tasks with high clinical complexity. Furthermore, GDPR compliance (Article 9 on health data) mandates rigorous pseudonymization of corpora before any external annotation, along with a data processing agreement meeting the regulation’s requirements.
Document NER
In document-centric contexts such as information extraction from contracts, invoices, or administrative forms, text annotation for NER frequently combines with layout recognition tasks. The entity is not only present in the text sequence but in a named field, a table cell, or a specifically positioned paragraph within the document structure. Document AI models such as LayoutLM and its variants train on corpora that integrate both textual content and the position coordinates of elements. The preparation of such corpora, which articulates NER with document classification, is examined in detail in our article on document classification through text annotation.
Multilingual NER
Text annotation for NER in languages other than English raises specific challenges that generic tools and schemas do not always address. French presents compound entities governed by language-specific orthographic rules: institutions such as “la Caisse nationale d’assurance maladie” or “le Conseil d’État” must be delimited according to an explicit convention on capitalization and determiners. Agglutinative languages like Turkish or Finnish present additional tokenization challenges, since a single word form can encode multiple pieces of morphological information. For multilingual projects, it is recommended to maintain a unified entity schema while adapting illustrative examples to each language and mobilizing native annotators capable of identifying terminological and cultural nuances that affect annotation decisions.
From annotated corpus to NER model: the complete pipeline
A text annotation project for NER does not end with corpus delivery. It is part of an iterative pipeline that keeps the annotation team and machine learning engineers interacting throughout the model development cycle. This iteration is not a sign of failure in the first pass: it is inherent to the nature of NLP projects and must be planned into the project’s budget and schedule from the outset.
Iterating on the schema
First model evaluations frequently reveal problematic entities: some types are poorly learned because corpus examples are too homogeneous or too sparse; others are confused because definitions do not allow the model to discriminate between them with confidence. These evaluation results must feed a schema revision and, where needed, a targeted re-annotation phase focused on underperforming entity types. Attempting to freeze the schema definitively before seeing the first model metrics is a frequent mistake in text annotation projects, one that significantly lengthens development cycles and generates corpora of which a substantial portion must be re-annotated.
Export formats and active learning
The most widely used text annotation tools for NER (Label Studio, Prodigy, Doccano) export in various formats: JSON-LINES, CoNLL, SpaCy native, or annotated JSONL. The machine learning pipeline must be able to ingest the annotation tool’s output format, or an explicit conversion step must be integrated and tested from the pilot batch onward, to avoid surprises at the first training attempt. Active learning approaches further optimize corpus coverage by prioritizing for annotation the documents on which the model is least confident, thereby reducing the total annotation volume needed to reach a target performance level.
NER and automation: what LLMs can and cannot do
Large language models have demonstrated NER capabilities in zero-shot mode on generic schemas. For common entity types such as PERSON, ORGANIZATION, or LOCATION, they produce exploitable results without a pre-existing annotated corpus. However, as soon as the schema becomes domain-specialized, the entities are domain-specific concepts, and the boundaries are governed by precise rules, LLM performance in automatic mode degrades significantly, and their errors are difficult to audit at scale. The hybrid approach, which uses an LLM for pre-annotation and human annotators for verification and correction, offers a sound balance between productivity and reliability on projects with moderately specialized schemas.
For high-stakes projects, particularly in healthcare, legal, and industrial domains, fully supervised text annotation by teams trained in the relevant domain remains the only guarantee of corpus consistency, traceability, and compliance. If your organization is considering building a NER corpus or evaluating the feasibility of an information extraction project, contact our team to discuss your sector-specific and regulatory requirements.
