Package index • formex

Detect & classify a column

The primary entry point. Route a column to its ontology type and reconcile value- and name-based evidence.

guess_data_type(): Guess the ontology-anchored data type of a column

guess_column(): Guess a column's type from its values and its name

score_data_type(): Score a column against every registered detector

classify_by_name(): Classify a column by its variable name

fx_classify_column(): Classify a single column under deployment rules (sample + gate + score)

fx_classify_df(): Classify every column of a data frame under deployment rules

Stabilize

Execute an ontology recipe to normalize a column to its stable form.

fx_stabilize(): Stabilize a column via its ontology recipe

fx_transform(): Run an ontology transform-DSL expression on a vector

The ontology

The shared vocabulary of semantic types, plus tooling to audit and evolve it.

fx_ontology(): The research data-type ontology (v7)

fx_validate_ontology(): Integrity-check an ontology table

fx_ontology_diff(): Classify the differences between two ontology versions

fx_ontology_impact_table(): Ontology change impact table

fx_impacted_cases(): Map an ontology diff onto banked cases

fx_migrate_bank(): Apply the mechanical half of a bank migration

fx_relabel_queue(): The relabel work list, highest-value first

fx_dgf_fields(): Format a type prediction as governance-file (DGF) fields

fx_triage_field(): Triage a column for banking as a test case

Column detectors

Deterministic value predicates, one per format. A column is typed by the share of its values a detector accepts.

is_abbr_month_date(): Is it a day-month-year date with an abbreviated month?

is_address(): Is it a full postal address?

is_asin(): Is it an Amazon ASIN?

is_bibcode(): Is it an ADS bibcode?

is_block_fips(): Is it a census block FIPS code?

is_block_group_fips(): Is it a census block-group FIPS code?

is_boolean(): Is it a boolean / indicator value?

is_ca_postal_code(): Is it a Canadian postal code (or FSA)?

is_cbsa(): Is it a CBSA code?

is_census_region(): Is it a US census region or division?

is_chinese_resident_id(): Is it a Chinese resident identity card number?

is_city_name(): Is it a US city / place name?

is_city_state(): Is it a "City, State" value?

is_clock_time(): Is it a clock time?

is_cmyk_color(): Is it a CMYK quadruple?

is_color_name(): Is it a named color?

is_coord_latlon(): Is it a latitude/longitude pair (latitude first)?

is_coord_lonlat(): Is it a longitude/latitude pair (longitude first)?

is_country_code(): Is it an ISO 3166-1 country code?

is_county_fips(): Is it a county FIPS code?

is_csa(): Is it a CSA code?

is_currency_amount(): Is it a monetary amount?

is_currency_code(): Is it an ISO 4217 currency code?

is_currency_code_symbol(): Is it a currency code + symbol + amount?

is_currency_symbol(): Is it a currency symbol + amount?

is_date(): Is it a date or timestamp?

is_date_range(): Is it a date range?

is_day_of_week(): Is it a day-of-week name?

is_dea(): Is it a DEA registration number?

is_doi(): Is it a DOI?

is_ein(): Is it a US Employer Identification Number (EIN)?

is_email(): Is it an email address?

is_ensembl_id(): Is it an Ensembl gene/transcript/protein ID?

is_first_name(): Is it a common US given (first) name?

is_fiscal_year(): Is it a fiscal (or explicitly labelled) year?

is_full_name(): Is it a person's full name?

is_geojson(): Is it a GeoJSON geometry?

is_guid(): Is it a GUID / UUID?

is_hex_color(): Is it a hex color?

is_hijri_date(): Is it a Hijri (Islamic-calendar) date?

is_hsl_color(): Is it an HSL triple?

is_http_status(): Is it an HTTP status code?

is_icd10_code(): Is it an ICD-10 code?

is_icd9_code(): Is it an ICD-9 code?

is_intl_phone(): Is it an international (+ prefixed) phone number?

is_ipv4(): Is it an IPv4 address?

is_ipv6(): Is it an IPv6 address?

is_isbn(): Is it an ISBN (10 or 13)?

is_iso6709(): Is it an ISO 6709 lat/lon pair?

is_iso_date(): Is it an ISO 8601 date?

is_iso_datetime(): Is it an ISO 8601 datetime?

is_issn(): Is it an ISSN?

is_json(): Is it a JSON object or array?

is_last_name(): Is it a common US surname (last name)?

is_long_date(): Is it a verbose English date?

is_mac(): Is it a MAC address?

is_md5(): Is it an MD5 digest?

is_metro_area_name(): Is it a metropolitan-area name?

is_month_day(): Is it a Month-Day value (no year)?

is_month_of_year(): Is it a month-of-year name?

is_month_year(): Is it a Month-Year value?

is_msisdn(): Is it an MSISDN (mobile subscriber number)?

is_naics(): Is it a NAICS industry code?

is_nces_locale(): Is it an NCES locale code?

is_npi(): Is it an NPI (US health provider)?

is_orcid(): Is it an ORCID iD?

is_phone(): Is it a US / NANP phone number?

is_po_box(): Is it a PO box?

is_quarter(): Is it a year-quarter?

is_rfc2822_datetime(): Is it an RFC 2822 / HTTP datetime?

is_rgb_color(): Is it an RGB triple?

is_roman(): Is it a Roman numeral?

is_sha1(): Is it a SHA-1 digest?

is_sha256(): Is it a SHA-256 digest?

is_snp_id(): Is it a dbSNP reference SNP ID?

is_ssn(): Is it a US Social Security Number?

is_state_fips(): Is it a state FIPS code?

is_state_zip(): Is it a "State ZIP" value?

is_street_address(): Is it a street address?

is_tags(): Is it a tag set (a delimited multi-value field)?

is_tract_fips(): Is it a census tract FIPS code?

is_uei(): Is it a SAM.gov Unique Entity Identifier (UEI)?

is_uk_postcode(): Is it a UK postcode?

is_uniprot_id(): Is it a UniProt accession?

is_unix_time(): Is it a Unix timestamp (seconds)?

is_urban_rural(): Is it an urban/rural designation?

is_url(): Is it a URL?

is_us_date(): Is it a US numeric date (M/D/YY)?

is_us_datetime(): Is it a US numeric datetime with AM/PM?

is_us_state(): Is it a US state abbreviation?

is_version(): Is it a version identifier?

is_wkb(): Is it Well-Known Binary (WKB / EWKB) geometry?

is_wkt_geometry(): Is it a WKT (Well-Known Text) geometry?

is_yyyymm(): Is it a compact YYYYMM year-month?

is_zip_code(): Is it a US ZIP code?

as_EIN(): Format a value as a canonical hyphenated EIN

as_mm(): Format a value as a zero-padded two-digit month

as_yyyy(): Format a value as a four-digit year

as_yyyymm(): Format a YYYYMM value as "YYYY-MM"

Feature extraction & signatures

The value-profile the routers are built on, and the lightweight signature primitives derived from it.

fx_extract_features(): Extract the value-profile feature vector for a column

fx_extract_features_df(): Extract features for every column of a data frame

fx_feature_manifest(): Map each feature to its block and cascade tier

fx_blocks_for_features(): Minimal set of feature blocks needed for a set of features

default_lexicons(): Default controlled-vocabulary lexicons for the lexicon feature block

fx_type_signature(): Reduce a column to its 7-number type signature

fx_signature_distance(): Weighted distance between two type signatures

fx_path_distance(): Taxonomic distance between two ontology paths

fx_sibling_report(): Surface confusable type pairs through the taxonomy and the data

SIGNATURE_WEIGHTS: Signature axis weights (balanced default)

WEIGHTS_FORMAT: Signature axis weights tuned for format comparisons

Test-case bank & auditing

fx_make_test_case(): Build a banked test case from a column

fx_index_bank(): Index a directory of banked test cases

fx_bank_coverage(): Coverage of a test-case bank by type and difficulty

fx_roundtrip_check(): Check whether stabilized values survive a read/write round trip

fx_audit_ontology(): Round-trip audit an ontology file

Benchmark & stress-test harness

Score guess_data_type() over the frozen test library at every ontology level, track accuracy and runtime over time, and stress-test robustness with a controlled grid of perturbations.

fx_benchmark_data(): Locate and load the frozen benchmark dataset

fx_benchmark(): Benchmark guess_data_type() over the frozen test library

fx_benchmark_compare(): Compare two benchmark runs (track progress over time)

fx_benchmark_confusion(): Confusion structure: what shape does "wrong" take?

fx_benchmark_summary(): Summarize a benchmark run

fx_coverage_gaps(): Coverage gaps: difficulty-aware flagging of underperforming detectors

fx_perturb(): Perturb a column along one stress dimension

fx_stress_grid(): Run the stress-test grid over the benchmark

fx_stress_summary(): Summarize the stress grid: accuracy decay per dimension and level

fx_contamination_curve(): Contamination breaking-point curve

Evaluation & AutoType reporting

evaluate_columns(): Evaluate formex predictions against a labeled column corpus

eval_summary(): Summarize an evaluation into overall and per-class metrics

feasibility_score(): Score one ontology assignment by feasibility

score_feasibility_table(): Score a table of predictions against a feasibility crosswalk

generate_autotype_negatives(): Generate negative examples for a mutation tier

generate_autotype_report(): Print an autotype error report for a results table

generate_autotype_test_cases(): Build a labelled test-case table for one data type

autotype_summary(): Summarize an autotype results table by mutation tier

autotype_summary_row(): One master-table row for a data type

build_autotype_results(): Run a detector against positives and generated negatives

Other

as_EIN(): Format a value as a canonical hyphenated EIN

as_mm(): Format a value as a zero-padded two-digit month

as_yyyy(): Format a value as a four-digit year

as_yyyymm(): Format a YYYYMM value as "YYYY-MM"

autotype_summary(): Summarize an autotype results table by mutation tier

autotype_summary_row(): One master-table row for a data type

build_autotype_results(): Run a detector against positives and generated negatives

classify_by_name(): Classify a column by its variable name

data_type_tests: Labelled test cases for data-type detectors

default_lexicons(): Default controlled-vocabulary lexicons for the lexicon feature block

evaluate_columns(): Evaluate formex predictions against a labeled column corpus

eval_summary(): Summarize an evaluation into overall and per-class metrics

feasibility_score(): Score one ontology assignment by feasibility

fx_apply_plan(): Apply a transform plan to values (v7, experimental)

fx_audit_ontology(): Round-trip audit an ontology file

fx_bank_coverage(): Coverage of a test-case bank by type and difficulty

fx_benchmark(): Benchmark guess_data_type() over the frozen test library

fx_benchmark_compare(): Compare two benchmark runs (track progress over time)

fx_benchmark_confusion(): Confusion structure: what shape does "wrong" take?

fx_benchmark_data(): Locate and load the frozen benchmark dataset

fx_benchmark_summary(): Summarize a benchmark run

fx_blocks_for_features(): Minimal set of feature blocks needed for a set of features

fx_classify_column(): Classify a single column under deployment rules (sample + gate + score)

fx_classify_df(): Classify every column of a data frame under deployment rules

fx_contamination_curve(): Contamination breaking-point curve

fx_coverage_gaps(): Coverage gaps: difficulty-aware flagging of underperforming detectors

fx_dgf_fields(): Format a type prediction as governance-file (DGF) fields

fx_extract_features(): Extract the value-profile feature vector for a column

fx_extract_features_df(): Extract features for every column of a data frame

fx_feature_manifest(): Map each feature to its block and cascade tier

fx_format(): A value's format descriptor (v7, experimental)

fx_impacted_cases(): Map an ontology diff onto banked cases

fx_index_bank(): Index a directory of banked test cases

fx_make_test_case(): Build a banked test case from a column

fx_migrate_bank(): Apply the mechanical half of a bank migration

fx_ontology(): The research data-type ontology (v7)

fx_ontology_diff(): Classify the differences between two ontology versions

fx_ontology_impact_table(): Ontology change impact table

fx_path_distance(): Taxonomic distance between two ontology paths

fx_perturb(): Perturb a column along one stress dimension

fx_plan_transform(): Plan a transform from one format to another (v7, experimental)

fx_relabel_queue(): The relabel work list, highest-value first

fx_roundtrip_check(): Check whether stabilized values survive a read/write round trip

fx_sibling_report(): Surface confusable type pairs through the taxonomy and the data

fx_signature_distance(): Weighted distance between two type signatures

fx_stabilize(): Stabilize a column via its ontology recipe

fx_stress_grid(): Run the stress-test grid over the benchmark

fx_stress_summary(): Summarize the stress grid: accuracy decay per dimension and level

fx_transform(): Run an ontology transform-DSL expression on a vector

fx_transformable(): Is one format reachable from another? (v7, experimental)

fx_transforms(): The transform registry (v7, experimental)

fx_triage_field(): Triage a column for banking as a test case

fx_type_signature(): Reduce a column to its 7-number type signature

fx_validate_ontology(): Integrity-check an ontology table

generate_autotype_negatives(): Generate negative examples for a mutation tier

generate_autotype_report(): Print an autotype error report for a results table

generate_autotype_test_cases(): Build a labelled test-case table for one data type

guess_column(): Guess a column's type from its values and its name

guess_data_type(): Guess the ontology-anchored data type of a column

is_abbr_month_date(): Is it a day-month-year date with an abbreviated month?

is_address(): Is it a full postal address?

is_asin(): Is it an Amazon ASIN?

is_bibcode(): Is it an ADS bibcode?

is_block_fips(): Is it a census block FIPS code?

is_block_group_fips(): Is it a census block-group FIPS code?

is_boolean(): Is it a boolean / indicator value?

is_ca_postal_code(): Is it a Canadian postal code (or FSA)?

is_cbsa(): Is it a CBSA code?

is_census_region(): Is it a US census region or division?

is_chinese_resident_id(): Is it a Chinese resident identity card number?

is_city_name(): Is it a US city / place name?

is_city_state(): Is it a "City, State" value?

is_clock_time(): Is it a clock time?

is_cmyk_color(): Is it a CMYK quadruple?

is_color_name(): Is it a named color?

is_coord_latlon(): Is it a latitude/longitude pair (latitude first)?

is_coord_lonlat(): Is it a longitude/latitude pair (longitude first)?

is_country_code(): Is it an ISO 3166-1 country code?

is_county_fips(): Is it a county FIPS code?

is_csa(): Is it a CSA code?

is_currency_amount(): Is it a monetary amount?

is_currency_code(): Is it an ISO 4217 currency code?

is_currency_code_symbol(): Is it a currency code + symbol + amount?

is_currency_symbol(): Is it a currency symbol + amount?

is_date(): Is it a date or timestamp?

is_date_range(): Is it a date range?

is_day_of_week(): Is it a day-of-week name?

is_dea(): Is it a DEA registration number?

is_doi(): Is it a DOI?

is_ein(): Is it a US Employer Identification Number (EIN)?

is_email(): Is it an email address?

is_ensembl_id(): Is it an Ensembl gene/transcript/protein ID?

is_first_name(): Is it a common US given (first) name?

is_fiscal_year(): Is it a fiscal (or explicitly labelled) year?

is_full_name(): Is it a person's full name?

is_geojson(): Is it a GeoJSON geometry?

is_guid(): Is it a GUID / UUID?

is_hex_color(): Is it a hex color?

is_hijri_date(): Is it a Hijri (Islamic-calendar) date?

is_hsl_color(): Is it an HSL triple?

is_http_status(): Is it an HTTP status code?

is_icd10_code(): Is it an ICD-10 code?

is_icd9_code(): Is it an ICD-9 code?

is_intl_phone(): Is it an international (+ prefixed) phone number?

is_ipv4(): Is it an IPv4 address?

is_ipv6(): Is it an IPv6 address?

is_isbn(): Is it an ISBN (10 or 13)?

is_iso6709(): Is it an ISO 6709 lat/lon pair?

is_iso_date(): Is it an ISO 8601 date?

is_iso_datetime(): Is it an ISO 8601 datetime?

is_issn(): Is it an ISSN?

is_json(): Is it a JSON object or array?

is_last_name(): Is it a common US surname (last name)?

is_long_date(): Is it a verbose English date?

is_mac(): Is it a MAC address?

is_md5(): Is it an MD5 digest?

is_metro_area_name(): Is it a metropolitan-area name?

is_month_day(): Is it a Month-Day value (no year)?

is_month_of_year(): Is it a month-of-year name?

is_month_year(): Is it a Month-Year value?

is_msisdn(): Is it an MSISDN (mobile subscriber number)?

is_naics(): Is it a NAICS industry code?

is_nces_locale(): Is it an NCES locale code?

is_npi(): Is it an NPI (US health provider)?

is_orcid(): Is it an ORCID iD?

is_phone(): Is it a US / NANP phone number?

is_po_box(): Is it a PO box?

is_quarter(): Is it a year-quarter?

is_rfc2822_datetime(): Is it an RFC 2822 / HTTP datetime?

is_rgb_color(): Is it an RGB triple?

is_roman(): Is it a Roman numeral?

is_sha1(): Is it a SHA-1 digest?

is_sha256(): Is it a SHA-256 digest?

is_snp_id(): Is it a dbSNP reference SNP ID?

is_ssn(): Is it a US Social Security Number?

is_state_fips(): Is it a state FIPS code?

is_state_zip(): Is it a "State ZIP" value?

is_street_address(): Is it a street address?

is_tags(): Is it a tag set (a delimited multi-value field)?

is_tract_fips(): Is it a census tract FIPS code?

is_uei(): Is it a SAM.gov Unique Entity Identifier (UEI)?

is_uk_postcode(): Is it a UK postcode?

is_uniprot_id(): Is it a UniProt accession?

is_unix_time(): Is it a Unix timestamp (seconds)?

is_urban_rural(): Is it an urban/rural designation?

is_url(): Is it a URL?

is_us_date(): Is it a US numeric date (M/D/YY)?

is_us_datetime(): Is it a US numeric datetime with AM/PM?

is_us_state(): Is it a US state abbreviation?

is_version(): Is it a version identifier?

is_wkb(): Is it Well-Known Binary (WKB / EWKB) geometry?

is_wkt_geometry(): Is it a WKT (Well-Known Text) geometry?

is_yyyymm(): Is it a compact YYYYMM year-month?

is_zip_code(): Is it a US ZIP code?

ontology: Research data-type ontology (v7), parsed

score_data_type(): Score a column against every registered detector

score_feasibility_table(): Score a table of predictions against a feasibility crosswalk

SIGNATURE_WEIGHTS: Signature axis weights (balanced default)

WEIGHTS_FORMAT: Signature axis weights tuned for format comparisons