Crawler documentation

About the InfluenceKit fetcher

InfluenceKit is reporting software for influencer marketing campaigns. When a customer adds a URL to one of their reports, we fetch that page once to read its link-preview metadata. This page documents that fetcher so you can identify it in your logs, allow it, or block it.


What it does

The fetcher requests a single URL and reads its OpenGraph, Twitter-card and oEmbed metadata — title, description, thumbnail, canonical URL. We use that to render a link preview card in the customer's report, the same way a link posted to Slack or LinkedIn renders a preview.

It reads the <head> of the document and any oEmbed endpoint the page declares. It does not execute JavaScript, submit forms, or attempt to reach anything behind a login.

When it runs

Only when an InfluenceKit customer adds a specific URL to their own report. That is the only trigger. In particular, the fetcher:

  • does not spider — it never follows links off the page it was given;
  • does not discover URLs on its own, from sitemaps, feeds, or anywhere else;
  • requests one URL per customer action, not a crawl of your site.

Results are cached for 24 hours, so repeated views of the same report do not generate repeated requests. Overall volume against any single domain is low.

User-agent

Every request carries this user-agent string:

InfluenceKit/1.0 (+https://www.influencekit.com/bot)

The version number may change as we ship updates. Match on InfluenceKit/ rather than the full string if you want a rule that survives version bumps.

IP addresses

Requests come from these IPv4 addresses:

5.161.182.142
5.161.47.50
5.161.123.201
178.156.136.245

Machine-readable copies of these lists — plain text, one address per line — are published at:

These are the authoritative lists. If we add capacity, they change before the change reaches your logs.

Verifying a request is really us

A user-agent string can be forged, so if you need certainty, do a forward-confirmed reverse DNS lookup. Our egress IPs reverse-resolve to a subdomain of influencekit.com; resolving that hostname forward must return the same IP you saw in the request.

$ dig +short -x 5.161.182.142
# → a hostname under influencekit.com

$ dig +short <that hostname>
# → 5.161.182.142

If both halves match, the request came from us. If they don't, it didn't — regardless of what the user-agent claims.

Allowing us

If your reports show broken or empty preview cards for pages on your site, we're most likely being blocked before we reach them. To allow us, add either the user-agent or the IP addresses above to your allowlist.

Most CDNs and WAFs have a control for this — Cloudflare, Akamai, Fastly, AWS WAF, DataDome, Imperva and Sucuri all support allowlisting by user-agent or by source IP. IP allowlisting is the stricter of the two and the one we'd recommend.

Blocking us

We honor robots.txt. To keep us out, add:

User-agent: InfluenceKit
Disallow: /

Blocking us means link previews for your pages render without a title, description or thumbnail in customer reports. Nothing else about your site is affected.

Contact

Questions, takedown requests, or a request that we stop fetching a specific domain — email help@influencekit.com. Include the domain and, if you have them, timestamps and the IPs you saw.