From: Daniel <me@sc1f1dan.com>

Hi folks -

I have a door idea and wonder if it's possible. I'd like to create something
of a very limited electronic flight book (EFB) in BBS form. As background, pretty much all software tools I use to do flight prep is done on my android tablet. Weather data, maps, etc are all downloaded from sources provided by
the government and displayed in the apps. Laws & Regulations come from ecfr.gov. Weather data from aviationweather.gov (awc). Aeronautical
Information Manual (AIM) from faa.gov. And etcetera.

Right now, my focus is on ecfr.gov. They have a limited rest API setup for
all matters of querying the database except for the actual regulation. By
that I mean, if I understand all the options correctly, I can't use the API
to pull up an individual regulation, like, 14 CFR part 91.109

If I attempt accessing it with lynx, I get this generic error:

"Due to aggressive automated scraping of FederalRegister.gov and eCFR.gov, programmatic access to these sites is limited to access to our extensive developer APIs."

It's funny how they call it extensive. They must be using a government
version of the english dictionary. It goes on:

"If you are human user receiving this message, we can add your IP address to
a set of IPs that can access FederalRegister.gov & eCFR.gov; complete the CAPTCHA (bot test) below and click "Request Access". This process will be necessary for each IP address you wish to access the site from, requests are valid for approximately one quarter (three months) after which the process
may need to be repeated."

I don't get this message on a normal javascript enabled browser. I surmise it views queries from non-javascript browsers as a curl-like command.

My suspicion is that the undocumented features of the API are used by the
site. So I wonder if such calls could be derived by looking at site's code.

Now, the door I have in mind, would list all the regulation parts suggested
for each particular rating. Select the regulation and it would pull the data for output. This would be a minimum feature for v1.0 of the door. Various
other queries would be added over time. And, of course, other data from
various sources would fold in too.

Here's the URL: www.ecfr.gov

Now, i considered downloading the entirety of the regulations and simply writing code to query it. But, these regs change all the time. New regs
appear. Some amended. Some abolished. So, it's an organic thing. I'd rather maintain API calls to be honest.

Lastly, this door idea would be implemented on my future BBS, focusing on aviation and radio related hobbies.

Thank you -

Daniel
--- NewsGate v1.0 gamma 2
* Origin: News Gate @ Net396 -Huntsville, AL - USA (1:396/4)

From: nospam.Jay.Harris@f664.n229.z1.fidonet.org (Jay Harris)

On 02 Jul 2024, Daniel said the following...

If I attempt accessing it with lynx, I get this generic error:

"Due to aggressive automated scraping of FederalRegister.gov and
eCFR.gov, programmatic access to these sites is limited to access to our extensive developer APIs."

I don't get this message on a normal javascript enabled browser. I
surmise it views queries from non-javascript browsers as a curl-like command.

I don't know if you can change the user agent in lynx, but try and see if it works with curl.

e.g:
agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"

curl -A "$agent" <url>


I've had to do this in the past when pulling information for weather, they stopped allowing curl, but simply changing the user agent allowed it to work.


Jay

.... Modem: What landscapers do to dem lawns

--- NewsGate v1.0 gamma 2
* Origin: News Gate @ Net396 -Huntsville, AL - USA (1:396/4)

From: Daniel <me@sc1f1dan.com>

nospam.Jay.Harris@f664.n229.z1.fidonet.org (Jay Harris) writes:

On 02 Jul 2024, Daniel said the following...

If I attempt accessing it with lynx, I get this generic error:

"Due to aggressive automated scraping of FederalRegister.gov and eCFR.gov, programmatic access to these sites is limited to access to our extensive developer APIs."

I don't get this message on a normal javascript enabled browser. I surmise it views queries from non-javascript browsers as a curl-like command.

I don't know if you can change the user agent in lynx, but try and see if it works with curl.

e.g:
agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"

curl -A "$agent" <url>

I've had to do this in the past when pulling information for weather, they stopped allowing curl, but simply changing the user agent allowed it to work.

Okay thanks. Do you have an opinion on the door idea by chance?
--- NewsGate v1.0 gamma 2
* Origin: News Gate @ Net396 -Huntsville, AL - USA (1:396/4)