# Scraping Data with Python

So I reached a point where I needed to get data from a 3rd party portal to send on to customers. 

I've got access to the backend but there's no API so making the data useful in a customer context requires some hacking. 

I ended up going with the Python Requests library and used Xpath

```
# create sesson
session_requests = requests.session()
# extract CSRF token using xpath and lxml
login_url = "https://example.com/login.php"
result = session_requests.get(login_url)
tree = html.fromstring(result.text)
authenticity_token = list(
set(tree.xpath("//input[@name='loginSubmit']/@value")))[0]

# login & send payload
result = session_requests.post(
login_url,
data=payload,
headers=dict(referer=login_url)
)
```
## Lessons learned
Load all the assets
- Interestingly it's important that you load all the assets (css and js) from the page to keep the scraper looking like a normal web client. Call it stealthy if you like.

- Useful for legacy integration.
Scraping if not used maliciously can actually save lots of integration time. When dealing with legacy systems managed by 3rd parties it's an indispensable technique. 

- Consider the resource you're scraping
Always consider if the 3rd party is ok with you doing this. In my case I have permission to use the information.

- Remember the CSRF token
CSRF tokens are a security measure to protect against bots. We're writing a scraping bot. But thankfully simpler forms are easy to submit - just include the CSRF token in the request after you identify and scrape it.

-Chasing a moving target
Sites that change structure frequently are harder to scrape consistently. There will always be times when scraping breaks and needs re-adjustment.

## Next Challenge
Scraping JS table data.