RFC 8805
A Format for Self-Published IP Geolocation Feeds
A Format for Self-Published IP Geolocation Feeds ✨
- RFC 8805 🎀 (ISSN 2070-1721)
- Category: Informational 🦋, an Independent Submission, so it's not an Internet Standards Track spec
- Date: August 2020 🌸
- Authors: E. Kline (Loon LLC), K. Duleba (Google), Z. Szamonek (Google), S. Moser (Google Switzerland GmbH), and W. Kumari (Google) 👑
- Obsoletes / Updates: this title block doesn't list any RFCs that it obsoletes or updates 🧚
Abstract
okay so this doc records a format where a network operator can publish a mapping of IP address prefixes to simplified geolocation info 🗺️ (colloquially a "geolocation feed" 💖). Interested parties can poll and parse these feeds to update or merge with other geolocation data sources and procedures. The format intentionally only allows coarse-level location, and that's the whole vibe 🫶.
Some technical orgs running networks that hop from one conference location to the next have already experimented with small geolocation feeds 🎪.
This doc describes a format that's already deployed in the wild 🔥. At least one consumer (Google) has plugged these feeds into a geolocation data pipeline, and a significant number of ISPs use it to tell everyone where their prefixes should be geolocated 💅.
Status of This Memo
This document is not an Internet Standards Track specification; it is published for informational purposes 📚.
This is a contribution to the RFC Series, independent of any other RFC stream 🌈. The RFC Editor chose to publish it at their discretion and makes no statement about its value for implementation or deployment. Documents approved for publication by the RFC Editor are not candidates for any level of Internet Standard, see Section 2 of RFC 7841 🙈.
Info about the current status of this document, any errata, and how to give feedback can be found at https://www.rfc-editor.org/info/rfc8805 ✨.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved ©️.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document 📜.
Table of Contents
1. Introduction
1.1. Motivation
1.2. Requirements Notation
1.3. Assumptions about Publication
2. Self-Published IP Geolocation Feeds
2.1. Specification
2.1.1. Geolocation Feed Individual Entry Fields
2.1.1.1. IP Prefix
2.1.1.2. Alpha2code (Previously: 'country')
2.1.1.3. Region
2.1.1.4. City
2.1.1.5. Postal Code
2.1.2. Prefixes with No Geolocation Information
2.1.3. Additional Parsing Requirements
2.2. Examples
3. Consuming Self-Published IP Geolocation Feeds
3.1. Feed Integrity
3.2. Verification of Authority
3.3. Verification of Accuracy
3.4. Refreshing Feed Information
4. Privacy Considerations
5. Relation to Other Work
6. Security Considerations
7. Planned Future Work
8. Finding Self-Published IP Geolocation Feeds
8.1. Ad Hoc 'Well-Known' URIs
8.2. Other Mechanisms
9. IANA Considerations
10. References
10.1. Normative References
10.2. Informative References
Appendix A. Sample Python Validation Code
Acknowledgements
Authors' Addresses1. Introduction 🌷
1.1. Motivation
okay so providers of services over the Internet have gotten really dependent on best-effort geolocation info to make the user experience better 🥰. Locality info can help direct traffic to the nearest serving location, guess the likely native language, and add context for services that deal with search queries 🔍.
When an ISP, for example, moves the location where an IP prefix is deployed, services that use geolocation info can start to suffer degraded performance 😭. That can lead to customer complaints, possibly straight to the ISP. Getting correct geolocation data out there is messy because there's no centralized way to coordinate and communicate geolocation info to every interested consumer 🫠.
This doc records a format where a network operator (an ISP, an enterprise, or any organization that thinks the geolocation of its IP prefixes is worth caring about) can publish a mapping of IP address prefixes to simplified geolocation info, aka a "geolocation feed" 💖. Interested parties can poll and parse these feeds to update or merge with other geolocation data sources and procedures.
This doc describes a format that's already deployed. At least one consumer (Google) has put these feeds into a geolocation data pipeline, and a significant number of ISPs are using it to tell everyone where their prefixes should be geolocated 🌈.
1.2. Requirements Notation
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎯.
As this is an informational document about a data format and a set of operational practices that are already in use, requirements notation captures the design goals of the authors and implementors. 🧁
1.3. Assumptions about Publication
This doc describes both a format and a mechanism for publishing data, on the assumption that the network operator who's been delegated operational responsibility for any published data wants it to be public 🌍. Any privacy risk is bounded by the format, and feed publishers MAY omit prefixes or any location field associated with a given prefix to further protect privacy (see Section 2.1 for details about exactly which fields may be omitted) 🙈. Feed publishers take on the responsibility of deciding which data should be made public 💅.
This doc does not include a mechanism to communicate acceptable use policies for self-published data. Publishing is inferred as the publisher wanting the data to be usefully consumed, kind of like how people publish host names, cryptographic keys, and Sender Policy Framework (SPF) records [RFC7208] in the DNS 🔐. It's giving "I want you to use this" energy 💜.
2. Self-Published IP Geolocation Feeds 🦋
The format described here was built to meet the needs of network operators who want to rapidly and usefully share geolocation info changes 🏃♀️. Originally there was a specific case where regional operators wanted to publish location changes instead of waiting for geolocation algorithms to "learn" about them 🤓. Later, technical conferences that often use the same network prefixes, advertised from different conference locations, experimented with publishing geolocation feeds ahead of network location changes so they could better serve attendees 🎪.
At its simplest, the mechanism is a network operator publishing a file (the "geolocation feed") with several text entries, one per line ✍️. Each entry is keyed by a unique (within the feed) IP prefix, or a single IP address, followed by a sequence of network locality attributes to be ascribed to the given prefix 🧚.
2.1. Specification
For operational simplicity, every feed should contain data about all IP addresses the provider wants to publish. Alternatives, like publishing only entries for IP addresses whose geolocation data has changed, or that differ from current observed geolocation behavior "at large", are likely to be too operationally complex 🤯.
Feeds MUST use UTF-8 [RFC3629] character encoding. Lines are delimited by a line break (CRLF) (as specified in [RFC4180]), and blank lines are ignored 🫧. Text from a '#' character to the end of the current line is treated as a comment only and is similarly ignored (note that this does not strictly follow [RFC4180], which has no support for comments) 💬.
Feed lines that are not comments MUST be formatted as comma-separated values (CSV), as described in [RFC4180]. Each feed entry is a text line of the form:
ip_prefix,alpha2code,region,city,postal_codeThe IP prefix field is REQUIRED, all others are OPTIONAL (can be empty), though the requisite minimum number of commas SHOULD be present 📏.
2.1.1. Geolocation Feed Individual Entry Fields ✨
2.1.1.1. IP Prefix
REQUIRED: Each IP prefix field MUST be either a single IP address or an IP prefix in Classless Inter-Domain Routing (CIDR) notation, in conformance with Section 3.1 of [RFC4632] for IPv4 or Section 2.3 of [RFC4291] for IPv6 🌐.
Examples include 192.0.2.1 and 192.0.2.0/24 for IPv4, and 2001:db8::1 and 2001:db8::/32 for IPv6 💖.
2.1.1.2. Alpha2code (Previously: 'country')
OPTIONAL: The alpha2code field, if non-empty, MUST be a 2-letter ISO country code conforming to ISO 3166-1 alpha 2 [ISO.3166.1alpha2]. Parsers SHOULD treat this field case-insensitively 🌏.
Earlier versions of this doc called this field "country", and it may still be referred to as such in existing tools and interfaces 🕰️.
Parsers MAY additionally support other 2-letter codes outside the ISO 3166-1 alpha 2 codes, such as the 2-letter codes from the "Exceptionally reserved codes" [ISO-GLOSSARY] set 🦄.
Examples include US for the United States, JP for Japan, and PL for Poland 🇺🇸🇯🇵🇵🇱.
2.1.1.3. Region
OPTIONAL: The region field, if non-empty, MUST be an ISO region code conforming to ISO 3166-2 [ISO.3166.2]. Parsers SHOULD treat this field case-insensitively 🗾.
Examples include ID-RI for the Riau province of Indonesia, and NG-RI for the Rivers province in Nigeria 🌴.
2.1.1.4. City
OPTIONAL: The city field, if non-empty, SHOULD be free UTF-8 text, excluding the comma (',') character 🏙️.
Examples include "Dublin", "New York", and "Sao Paulo" (specifically "S" followed by 0xc3, 0xa3, and "o Paulo") 🌆.
2.1.1.5. Postal Code
OPTIONAL, DEPRECATED: The postal code field, if non-empty, SHOULD be free UTF-8 text, excluding the comma (',') character. The use of this field is deprecated. Consumers of feeds should be able to parse feeds containing these fields, but new feeds SHOULD NOT include this field due to the granularity of this information. See Section 4 for additional discussion 🚫.
Heads up bestie: this field is deprecated, so new feeds SHOULD NOT include it, but consumers still need to be able to parse it. ⚠️
Examples include 106-6126 (in Minato ward, Tokyo, Japan) 🗼💅
2.1.2. Prefixes with No Geolocation Information
okay so sometimes a publisher wants a prefix to have no location info at all 🙈 maybe it's reserved, not allocated or deployed yet, or it's being moved somewhere else. From the publisher's side, the old location info can safely be tossed 🗑️ and that's totally valid.
To say this, you just leave every field that would hold location info completely blank. For example:
192.0.2.0/24,,,,
2001:db8:1::/48,,,,
2001:db8:2::/48,,,,Historically, the user-assigned alpha2code ZZ was used for this exact same job 💅. It's not necessarily the preferred move, though, and no specific meaning is currently defined for any of the other user-assigned alpha2code values.
2.1.3. Additional Parsing Requirements
bestie, this part is the rulebook for parsers 📖✨
Feed entries that don't have an IP address or prefix field, or that have one that won't parse correctly, MUST be discarded. No exceptions 💀
Publishers SHOULD follow [RFC5952] for IPv6 prefix fields, but consumers MUST nevertheless accept all valid string representations. So be chill and accept every valid spelling 🫶
Duplicate IP address or prefix entries MUST be considered an error, and consumer implementations SHOULD log the repeated entries for further administrative review. Publishers SHOULD take measures to make sure there is one and only one entry per IP address and prefix. Main character energy is having exactly one entry per address 👑
Multiple entries that constitute nested prefixes are permitted. Consumers SHOULD consider the entry with the longest matching prefix (i.e., the "most specific") to be the best matching entry for a given IP address. The most specific one wins, periodt 🎀
Feed entries with non-empty optional fields that fail to parse, either in part or in full, SHOULD be discarded. It is RECOMMENDED that they also be logged for further administrative review 🌸
For compatibility with future additional fields, a parser MUST ignore any fields beyond those it expects. The data from the fields that are expected and that parse successfully MUST still be considered valid. Per Section 7, no extensions to this format are in use, nor are any anticipated ✨
2.2. Examples
Here are some example entries using different IP address formats, describing locations at alpha2code ("country code"), region, and city granularity levels, respectively 🌍
192.0.2.0/25,US,US-AL,,
192.0.2.5,US,US-AL,Alabaster,
192.0.2.128/25,PL,PL-MZ,,
2001:db8::/32,PL,,,
2001:db8:cafe::/48,PL,PL-MZ,,The IETF network publishes geolocation info for the meeting prefixes 🧚. It generally just comments out the last meeting's info and appends the new meeting's info. At the time of writing, the [GEO_IETF] contains:
# IETF106 (Singapore) - November 2019 - Singapore, SG
130.129.0.0/16,SG,SG-01,Singapore,
2001:df8::/32,SG,SG-01,Singapore,
31.133.128.0/18,SG,SG-01,Singapore,
31.130.224.0/20,SG,SG-01,Singapore,
2001:67c:1230::/46,SG,SG-01,Singapore,
2001:67c:370::/48,SG,SG-01,Singapore,Experimentally, RIPE has published geolocation info for their conference network prefixes, which change location with each new event 🦋 At the time of writing, [GEO_RIPE_NCC] contains:
193.0.24.0/21,NL,NL-ZH,Rotterdam,
2001:67c:64::/48,NL,NL-ZH,Rotterdam,Similarly, ICANN has published geolocation info for their portable conference network prefixes 🌈 At the time of writing, [GEO_ICANN] contains:
199.91.192.0/21,MA,MA-07,Marrakech
2620:f:8000::/48,MA,MA-07,MarrakechA longer example is the [GEO_Google] Google Corp Geofeed, which lists geolocation info for Google's corporate offices 🏢 At the time of writing, Google processes approximately 400 feeds comprising more than 750,000 IPv4 and IPv6 prefixes. That is a LOT of prefixes 😭
3. Consuming Self-Published IP Geolocation Feeds
Consumers MAY treat published feed data as a hint only, and MAY choose to prefer other sources of geolocation information for any given IP prefix 🫧 No matter where a consumer stands on a given feed, there are still some points of note for sensibly and effectively consuming published feeds.
3.1. Feed Integrity
The integrity of published info SHOULD be protected by securing the means of publication, for example by using HTTP over TLS [RFC2818] 🔐 Whenever possible, consumers SHOULD prefer retrieving geolocation feeds in a way that guarantees the feed's integrity. Trust, but make sure the feed hasn't been messed with 🫶
3.2. Verification of Authority
Consumers of self-published IP geolocation feeds SHOULD perform some form of verification that the publisher is actually authoritative for the addresses in the feed 🕵️♀️ The actual means of verification probably depends on how the feed was discovered. Ad hoc shared URIs, for example, will likely need an ad hoc verification process. Future automated means of feed discovery SHOULD come with an accompanying automated means of verification.
A consumer should only trust geolocation info for IP addresses or prefixes where the publisher has been verified as administratively authoritative. All other geolocation feed entries should be ignored and logged for further administrative review 🚫
3.3. Verification of Accuracy
Errors and inaccuracies may occur at many levels, and publishing and consuming geolocation data are no exceptions 🙃 To the extent practical, consumers SHOULD take steps to verify the accuracy of published locality. How you verify, how you resolve discrepancies, and which alternative sources you prefer are all left up to the feed consumer 💖
Consumers SHOULD decide on discrepancy thresholds, and SHOULD flag feed entries that exceed those thresholds for administrative review 🚩
3.4. Refreshing Feed Information
A publisher can change geolocation data at any time, and without notification, so consumers SHOULD implement mechanisms that periodically refresh local copies of feed data 🔄 In the absence of any other refresh timing info, it is recommended that consumers SHOULD refresh feeds no less often than weekly, and no more often than is likely to cause issues for the publisher. Balance is everything ⚖️
For feeds available via HTTPS (or HTTP), the publisher MAY communicate refresh timing info through the standard HTTP expiration model ([RFC7234]) ⏰ Specifically, publishers can include either an Expires header (Section 5.3 of [RFC7234]) or a Cache-Control header (Section 5.2 of [RFC7234]) specifying the max-age. Where practical, consumers SHOULD refresh feed info before the expiry time is reached.
4. Privacy Considerations
Publishers of geolocation feeds are advised to have fully considered any and all privacy implications of disclosing this info to the users of the described networks before publication 🕊️ A thorough understanding of the security considerations (Section 13 of [RFC6772]) of a chosen geolocation policy is highly recommended, including some of the limitations of information obscurity (Section 13.5 of [RFC6772]) (see also [RFC6772]). This is not a vibe to skip 😬
As noted in Section 2.1, each location field in an entry is optional, so publishers can express only the level of specificity they've deemed acceptable. There's no requirement that the level of specificity be consistent across all entries within a feed 🌙 In particular, the Postal Code field (Section 2.1.1.5) can provide very specific geolocation, sometimes down to within a building 🏠 Such specific Postal Code values MUST NOT be published in geofeeds without the express consent of the parties being located. Consent first, always 🫶
Operators who publish geolocation info are strongly encouraged to tell affected users and customers that this is happening, along with the potential privacy-related consequences and trade-offs 💜
5. Relation to Other Work
While this work was not originally done in conjunction with the GEOPRIV Working Group [GEOPRIV], Richard Barnes observed that it is nevertheless consistent with what the group has defined, both for address format and for privacy 🧠 The data elements in geolocation feeds are equivalent to the following XML structure ([RFC5139] [W3C.REC-xml-20081126]):
<civicAddress>
<country>country</country>
<A1>region</A1>
<A2>city</A2>
<PC>postal_code</PC>
</civicAddress>Providing geolocation info at this granularity is equivalent to the following privacy policy (the definition of the 'building' Section 6.5.1 of [RFC6772] level of disclosure) 🌸
<ruleset>
<rule>
<conditions/>
<actions/>
<transformations>
<provide-location profile="civic-transformation">
<provide-civic>building</provide-civic>
</provide-location>
</transformations>
</rule>
</ruleset>6. Security Considerations
As there is no true security in the obscurity of the location of any given IP address, self-publication of this data fundamentally opens no new attack vectors 🦹 For publishers, though, self-published data may make it easier to exploit such location data. It can, for example, make it easy to discover which prefixes are populated with customers as distinct from prefixes not generally in use 👀
For consumers, feed retrieval processes may receive input from potentially hostile sources (e.g., in the event of hijacked traffic). This is a big deal 😱 As such, proper input validation and defense measures MUST be taken (see the discussion in Section 3.1).
Similarly, consumers who do not perform sufficient verification of published data bear the same risks as from other forms of geolocation configuration errors (see the discussion in Sections 3.2 and 3.3) 💀
Validation of a feed's contents includes verifying that the publisher is authoritative for the IP prefixes included in the feed. Failure to verify IP prefix authority would, for example, allow ISP Bob to make geolocation statements about IP space held by ISP Alice. Yikes 🙀 At this time, only out-of-band verification methods are implemented (i.e., an ISP's feed may be verified against publicly available IP allocation data).
7. Planned Future Work
To more flexibly support future extensions, using a more expressive feed format has been suggested 🔮 Use of JavaScript Object Notation (JSON) [RFC8259] in particular has been discussed. However, at the time of writing, no such specification or implementation exists. Work on extensions is deferred until a more suitable format has been selected ⏳
The authors are planning on writing a document describing such a new format. This document describes a currently deployed and used format. Given the extremely limited extensibility of the present format, no extensions to it are anticipated 🙅♀️ Extensibility requirements are instead expected to be integral to the development of a new format 🧩
8. Finding Self-Published IP Geolocation Feeds
The issue of finding, and later verifying, geolocation feeds is not formally specified in this document 🔍 At this time, only ad hoc feed discovery and verification has a modicum of established practice (see below). Discussion of other mechanisms has been removed for clarity ✨
8.1. Ad Hoc 'Well-Known' URIs 🎀
okay so, until now geolocation feeds have mostly been passed around informally as HTTPS URIs that people drop into email threads 📧✨ Three example URIs ([GEO_IETF], [GEO_RIPE_NCC], and [GEO_ICANN]) show this. They describe networks whose locations change over time, and the people who run them and how they run them are well known in their own technical communities 🫶
checking the feeds is just as ad hoc 💅 People usually verify them using:
- personal knowledge of the people involved in the exchange, and
- comparing the prefixes listed in the feed with the BGP-advertised prefixes of Autonomous System Numbers that the publishers are known to operate.
ad hoc stuff is lowkey great for early experiments by producers and consumers 🧚 but it's giving "not built for the long haul" 💀 It won't hold up for long-term, widespread use by lots of different parties. So future versions of any self-published geolocation feed mechanism SHOULD deal with scalability by defining a way to discover and verify the operational authority over advertised prefixes automatically 🔥 that's the main character glow-up we need 👑
8.2. Other Mechanisms 🌸
tea time 🍵 earlier versions of this document mentioned using the WHOIS service [RFC3912], which is run by Regional Internet Registries (RIRs), and some possible DNS-based schemes for discovering and validating geofeeds. As far as the authors know, nobody has ever actually implemented those mechanisms 😭 so that speculative text has been removed to avoid any confusion 🫶
9. IANA Considerations ✨
this one's short and sweet 💖 This document has no IANA actions.
10. References 🌈
10.1. Normative References 💜
[ISO.3166.1alpha2]
ISO, "ISO 3166-1 decoding table",
<http://www.iso.org/iso/home/standards/country_codes/iso-
3166-1_decoding_table.htm>.
[ISO.3166.2]
ISO, "ISO 3166-2:2007",
<http://www.iso.org/iso/home/standards/
country_codes.htm#2012_iso3166-2>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, DOI 10.17487/RFC3629, November
2003, <https://www.rfc-editor.org/info/rfc3629>.
[RFC4180] Shafranovich, Y., "Common Format and MIME Type for Comma-
Separated Values (CSV) Files", RFC 4180,
DOI 10.17487/RFC4180, October 2005,
<https://www.rfc-editor.org/info/rfc4180>.
[RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing
Architecture", RFC 4291, DOI 10.17487/RFC4291, February
2006, <https://www.rfc-editor.org/info/rfc4291>.
[RFC4632] Fuller, V. and T. Li, "Classless Inter-domain Routing
(CIDR): The Internet Address Assignment and Aggregation
Plan", BCP 122, RFC 4632, DOI 10.17487/RFC4632, August
2006, <https://www.rfc-editor.org/info/rfc4632>.
[RFC5952] Kawamura, S. and M. Kawashima, "A Recommendation for IPv6
Address Text Representation", RFC 5952,
DOI 10.17487/RFC5952, August 2010,
<https://www.rfc-editor.org/info/rfc5952>.
[RFC7234] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
Ed., "Hypertext Transfer Protocol (HTTP/1.1): Caching",
RFC 7234, DOI 10.17487/RFC7234, June 2014,
<https://www.rfc-editor.org/info/rfc7234>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[W3C.REC-xml-20081126]
Bray, T., Paoli, J., Sperberg-McQueen, M., Maler, E., and
F. Yergeau, "Extensible Markup Language (XML) 1.0 (Fifth
Edition)", World Wide Web Consortium Recommendation REC-
xml-20081126, November 2008,
<http://www.w3.org/TR/2008/REC-xml-20081126>.10.2. Informative References 🦋
[GEOPRIV] IETF, "Geographic Location/Privacy (geopriv)",
<http://datatracker.ietf.org/wg/geopriv/>.
[GEO_Google]
Google, LLC, "Google Corp Geofeed",
<https://www.gstatic.com/geofeed/corp_external>.
[GEO_ICANN]
ICANN, "ICANN Meeting Geolocation Data",
<https://meeting-services.icann.org/geo/google.csv>.
[GEO_IETF] Kumari, W., "IETF Meeting Network Geolocation Data",
<https://noc.ietf.org/geo/google.csv>.
[GEO_RIPE_NCC]
Schepers, M., "RIPE NCC Meeting Geolocation Data",
<https://meetings.ripe.net/geo/google.csv>.
[IPADDR_PY]
Shields, M. and P. Moody, "Google's Python IP address
manipulation library",
<http://code.google.com/p/ipaddr-py/>.
[ISO-GLOSSARY]
ISO, "Glossary for ISO 3166",
<https://www.iso.org/glossary-for-iso-3166.html>.
[RFC2818] Rescorla, E., "HTTP Over TLS", RFC 2818,
DOI 10.17487/RFC2818, May 2000,
<https://www.rfc-editor.org/info/rfc2818>.
[RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912,
DOI 10.17487/RFC3912, September 2004,
<https://www.rfc-editor.org/info/rfc3912>.
[RFC5139] Thomson, M. and J. Winterbottom, "Revised Civic Location
Format for Presence Information Data Format Location
Object (PIDF-LO)", RFC 5139, DOI 10.17487/RFC5139,
February 2008, <https://www.rfc-editor.org/info/rfc5139>.
[RFC6772] Schulzrinne, H., Ed., Tschofenig, H., Ed., Cuellar, J.,
Polk, J., Morris, J., and M. Thomson, "Geolocation Policy:
A Document Format for Expressing Privacy Preferences for
Location Information", RFC 6772, DOI 10.17487/RFC6772,
January 2013, <https://www.rfc-editor.org/info/rfc6772>.
[RFC7208] Kitterman, S., "Sender Policy Framework (SPF) for
Authorizing Use of Domains in Email, Version 1", RFC 7208,
DOI 10.17487/RFC7208, April 2014,
<https://www.rfc-editor.org/info/rfc7208>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>.Appendix A. Sample Python Validation Code ✨
Here's a little Python validator for self-published ipgeo feeds 🐍💖 It reads CSV data in that format from standard input and runs basic checks on it. It's meant for feed publishers to run before they launch a feed 🚀 Quick heads-up: it doesn't check that every IP prefix is unique across the whole feed. It only checks the syntax of each line on its own. A complete validator MUST also ensure IP prefix uniqueness 👀
The main source file is "ipgeo_feed_validator.py" 🎀 It needs the open source ipaddr Python library to parse and validate IP addresses and CIDR prefixes [IPADDR_PY].
#!/usr/bin/python
#
# Copyright (c) 2012 IETF Trust and the persons identified as
# authors of the code. All rights reserved. Redistribution and use
# in source and binary forms, with or without modification, is
# permitted pursuant to, and subject to the license terms contained
# in, the Simplified BSD License set forth in Section 4.c of the
# IETF Trust's Legal Provisions Relating to IETF
# Documents (http://trustee.ietf.org/license-info).
"""Simple format validator for self-published ipgeo feeds.
This tool reads CSV data in the self-published ipgeo feed format
from the standard input and performs basic validation. It is
intended for use by feed publishers before launching a feed.
"""
import csv
import ipaddr
import re
import sys
class IPGeoFeedValidator(object):
def __init__(self):
self.prefixes = {}
self.line_number = 0
self.output_log = {}
self.SetOutputStream(sys.stderr)
def Validate(self, feed):
"""Check validity of an IPGeo feed.
Args:
feed: iterable with feed lines
"""
for line in feed:
self._ValidateLine(line)
def SetOutputStream(self, logfile):
"""Controls where the output messages go do (STDERR by default).
Use None to disable logging.
Args:
logfile: a file object (e.g., sys.stdout) or None.
"""
self.output_stream = logfile
def CountErrors(self, severity):
"""How many ERRORs or WARNINGs were generated."""
return len(self.output_log.get(severity, []))
############################################################
def _ValidateLine(self, line):
line = line.rstrip('\r\n')
self.line_number += 1
self.line = line.split('#')[0]
self.is_correct_line = True
if self._ShouldIgnoreLine(line):
return
fields = [field for field in csv.reader([line])][0]
self._ValidateFields(fields)
self._FlushOutputStream()
def _ShouldIgnoreLine(self, line):
line = line.strip()
if line.startswith('#'):
return True
return len(line) == 0
############################################################
def _ValidateFields(self, fields):
assert(len(fields) > 0)
is_correct = self._IsIPAddressOrPrefixCorrect(fields[0])
if len(fields) > 1:
if not self._IsAlpha2CodeCorrect(fields[1]):
is_correct = False
if len(fields) > 2 and not self._IsRegionCodeCorrect(fields[2]):
is_correct = False
if len(fields) != 5:
self._ReportWarning('5 fields were expected (got %d).'
% len(fields))
############################################################
def _IsIPAddressOrPrefixCorrect(self, field):
if '/' in field:
return self._IsCIDRCorrect(field)
return self._IsIPAddressCorrect(field)
def _IsCIDRCorrect(self, cidr):
try:
ipprefix = ipaddr.IPNetwork(cidr)
if ipprefix.network._ip != ipprefix._ip:
self._ReportError('Incorrect IP Network.')
return False
if ipprefix.is_private:
self._ReportError('IP Address must not be private.')
return False
except:
self._ReportError('Incorrect IP Network.')
return False
return True
def _IsIPAddressCorrect(self, ipaddress):
try:
ip = ipaddr.IPAddress(ipaddress)
except:
self._ReportError('Incorrect IP Address.')
return False
if ip.is_private:
self._ReportError('IP Address must not be private.')
return False
return True
############################################################
def _IsAlpha2CodeCorrect(self, alpha2code):
if len(alpha2code) == 0:
return True
if len(alpha2code) != 2 or not alpha2code.isalpha():
self._ReportError(
'Alpha 2 code must be in the ISO 3166-1 alpha 2 format.')
return False
return True
def _IsRegionCodeCorrect(self, region_code):
if len(region_code) == 0:
return True
if '-' not in region_code:
self._ReportError('Region code must be in ISO 3166-2 format.')
return False
parts = region_code.split('-')
if not self._IsAlpha2CodeCorrect(parts[0]):
return False
return True
############################################################
def _ReportError(self, message):
self._ReportWithSeverity('ERROR', message)
def _ReportWarning(self, message):
self._ReportWithSeverity('WARNING', message)
def _ReportWithSeverity(self, severity, message):
self.is_correct_line = False
output_line = '%s: %s\n' % (severity, message)
if severity not in self.output_log:
self.output_log[severity] = []
self.output_log[severity].append(output_line)
if self.output_stream is not None:
self.output_stream.write(output_line)
def _FlushOutputStream(self):
if self.is_correct_line: return
if self.output_stream is None: return
self.output_stream.write('line %d: %s\n\n'
% (self.line_number, self.line))
############################################################
def main():
feed_validator = IPGeoFeedValidator()
feed_validator.Validate(sys.stdin)
if feed_validator.CountErrors('ERROR'):
sys.exit(1)
if __name__ == '__main__':
main()And the unit test file, "ipgeo_feed_validator_test.py", comes along too 🧪 It gives basic coverage of the code above. One gap to know about: it does not test how non-ASCII UTF-8 strings are handled 🙈
#!/usr/bin/python
#
# Copyright (c) 2012 IETF Trust and the persons identified as
# authors of the code. All rights reserved. Redistribution and use
# in source and binary forms, with or without modification, is
# permitted pursuant to, and subject to the license terms contained
# in, the Simplified BSD License set forth in Section 4.c of the
# IETF Trust's Legal Provisions Relating to IETF
# Documents (http://trustee.ietf.org/license-info).
import sys
from ipgeo_feed_validator import IPGeoFeedValidator
class IPGeoFeedValidatorTest(object):
def __init__(self):
self.validator = IPGeoFeedValidator()
self.validator.SetOutputStream(None)
self.successes = 0
self.failures = 0
def Run(self):
self.TestFeedLine('# asdf', 0, 0)
self.TestFeedLine(' ', 0, 0)
self.TestFeedLine('', 0, 0)
self.TestFeedLine('asdf', 1, 1)
self.TestFeedLine('asdf,US,,,', 1, 0)
self.TestFeedLine('aaaa::,US,,,', 0, 0)
self.TestFeedLine('zzzz::,US', 1, 1)
self.TestFeedLine(',US,,,', 1, 0)
self.TestFeedLine('55.66.77', 1, 1)
self.TestFeedLine('55.66.77.888', 1, 1)
self.TestFeedLine('55.66.77.asdf', 1, 1)
self.TestFeedLine('2001:db8:cafe::/48,PL,PL-MZ,,02-784', 0, 0)
self.TestFeedLine('2001:db8:cafe::/48', 0, 1)
self.TestFeedLine('55.66.77.88,PL', 0, 1)
self.TestFeedLine('55.66.77.88,PL,,,', 0, 0)
self.TestFeedLine('55.66.77.88,,,,', 0, 0)
self.TestFeedLine('55.66.77.88,ZZ,,,', 0, 0)
self.TestFeedLine('55.66.77.88,US,,,', 0, 0)
self.TestFeedLine('55.66.77.88,USA,,,', 1, 0)
self.TestFeedLine('55.66.77.88,99,,,', 1, 0)
self.TestFeedLine('55.66.77.88,US,US-CA,,', 0, 0)
self.TestFeedLine('55.66.77.88,US,USA-CA,,', 1, 0)
self.TestFeedLine('55.66.77.88,USA,USA-CA,,', 2, 0)
self.TestFeedLine('55.66.77.88,US,US-CA,Mountain View,', 0, 0)
self.TestFeedLine('55.66.77.88,US,US-CA,Mountain View,94043',
0, 0)
self.TestFeedLine('55.66.77.88,US,US-CA,Mountain View,94043,'
'1600 Ampthitheatre Parkway', 0, 1)
self.TestFeedLine('55.66.77.0/24,US,,,', 0, 0)
self.TestFeedLine('55.66.77.88/24,US,,,', 1, 0)
self.TestFeedLine('55.66.77.88/32,US,,,', 0, 0)
self.TestFeedLine('55.66.77/24,US,,,', 1, 0)
self.TestFeedLine('55.66.77.0/35,US,,,', 1, 0)
self.TestFeedLine('172.15.30.1,US,,,', 0, 0)
self.TestFeedLine('172.28.30.1,US,,,', 1, 0)
self.TestFeedLine('192.167.100.1,US,,,', 0, 0)
self.TestFeedLine('192.168.100.1,US,,,', 1, 0)
self.TestFeedLine('10.0.5.9,US,,,', 1, 0)
self.TestFeedLine('10.0.5.0/24,US,,,', 1, 0)
self.TestFeedLine('fc00::/48,PL,,,', 1, 0)
self.TestFeedLine('fe00::/48,PL,,,', 0, 0)
print ('%d tests passed, %d failed'
% (self.successes, self.failures))
def IsOutputLogCorrectAtSeverity(self, severity,
expected_msg_count):
msg_count = self.validator.CountErrors(severity)
if msg_count != expected_msg_count:
print ('TEST FAILED: %s\nexpected %d %s[s], observed %d\n%s\n'
% (self.validator.line, expected_msg_count, severity,
msg_count,
str(self.validator.output_log[severity])))
return False
return True
def IsOutputLogCorrect(self, new_errors, new_warnings):
retval = True
if not self.IsOutputLogCorrectAtSeverity('ERROR', new_errors):
retval = False
if not self.IsOutputLogCorrectAtSeverity('WARNING',
new_warnings):
retval = False
return retval
def TestFeedLine(self, line, warning_count, error_count):
self.validator.output_log['WARNING'] = []
self.validator.output_log['ERROR'] = []
self.validator._ValidateLine(line)
if not self.IsOutputLogCorrect(warning_count, error_count):
self.failures += 1
return False
self.successes += 1
return True
if __name__ == '__main__':
IPGeoFeedValidatorTest().Run()Acknowledgements 🫶
The authors want to give a huge thank-you 💜 to the reviewers and early implementers, including but not limited to Mikael Abrahamsson, Andrew Alston, Ray Bellis, John Bond, Alissa Cooper, Andras Erdei, Stephen Farrell, Marco Hogewoning, Mike Joseph, Maciej Kuzniar, George Michaelson, Menno Schepers, Justyna Sidorska, Pim van Pelt, and Bjoern A. Zeeb 🌸
Special shoutout 👑 to Richard L. Barnes and Andy Newton, who contributed substantial review, text, and advice 😭🔥
Authors' Addresses 💌
okay bestie, this is the squad list 🫶 here's where you can find each of the five people who wrote this, with their mailing addresses and emails. Erik is at Loon LLC, and Krzysztof, Zoltan, Stefan, and Warren are all at Google, with Zoltan and Stefan based at Google Switzerland GmbH in Zürich. the whole crew is giving brilliant networking brain energy 🧚✨
Erik Kline 🦋
Erik Kline
Loon LLC
1600 Amphitheatre Parkway
Mountain View, CA 94043
United States of America
Email: ek@loon.comKrzysztof Duleba 🌸
Krzysztof Duleba
Google
1600 Amphitheatre Parkway
Mountain View, CA 94043
United States of America
Email: kduleba@google.comZoltan Szamonek 🎀
Zoltan Szamonek
Google Switzerland GmbH
Brandschenkestrasse 110
CH-8002 Zürich
Switzerland
Email: zszami@google.comStefan Moser 💜
Stefan Moser
Google Switzerland GmbH
Brandschenkestrasse 110
CH-8002 Zürich
Switzerland
Email: smoser@google.comWarren Kumari 🍓
Warren Kumari
Google
1600 Amphitheatre Parkway
Mountain View, CA 94043
United States of America
Email: warren@kumari.netthat's the whole document, slay 🌈👑 💅
📦 where this page comes from
- ietf: rfc8805 2020-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms