Automation Tool Use Deviation from AI-Related Tools Confirms Possible AI Hype Cycle Focus on Automation; Trend Now Reversing

Author(s): Jonathan Bennion

Originally published on Towards AI.

Trends suggest automation tools faced challenges during AI hype cycle

TLDR:

• Automation tools (Zapier as an example) API public development declined (-13.1% y/y) until last month, while AI-related APIs have experienced steady growth (+12.0% y/y) during same timeframe.
• Zapier’s recent spike may indicate strategic adaptation or solution to AI trends, with highest correlation to UIPath’s free tools, but correlation doesn’t equal causation either way.
• Caveat on public developer activity, so not accounting for private trends (which could be substantially different).

Question this quick analysis answers:

Did AI hype-infused solutions to workflow automation affect trends with Zapier’s workflow automation solutions, and could that be shaking out differently at an inflection point in the hype cycle?

Let’s start by importing the necessary libraries and loading our data (see my previous blog post for public development trend query out of GCP). Note this code is on my github repo in the form of a notebook.

# imports
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import numpy as np

# Load the data - in this case sourced from same query over weekend
data = pd.read_csv('ff.csv')

Long table format, so transformations are called for

# Convert 'month' to datetime
data['month'] = pd.to_datetime(data['month'])

# Filter out September 2024 - incomplete month
data = data[data['month'] < '2024-09-01']

# Filter data for the complete years (2023 and 2024)
data = data[data['month'].dt.year.isin([2023, 2024])]

# Separate Zapier data
zapier_data = data[data['keyword_category'] == 'zapier'].set_index('month')

# Aggregate all other categories as 'AI-related APIs'
ai_apis_data = data[data['keyword_category'] != 'zapier'].groupby('month')['new_repo_count'].sum().reset_index()
ai_apis_data = ai_apis_data.set_index('month')

# Calculate 7-day rolling average for smoothing
zapier_data['rolling_avg'] = zapier_data['new_repo_count'].rolling(window=7).mean()
ai_apis_data['rolling_avg'] = ai_apis_data['new_repo_count'].rolling(window=7).mean()

Zapier data I’d queried is so small (!) so the month-over-month variation isn’t going to lend to anything stat sig, by month, but in aggregate it’s likely going to help support a hypothesis (plotted with and without the below CI, ended up removing for legibility).

# Calculate 95% confidence intervals
def calculate_ci(data):
 confidence = 0.95
 degrees_of_freedom = len(data) - 1
 sample_mean = np.mean(data)
 sample_standard_error = stats.sem(data)

 ci = stats.t.interval(confidence=confidence,
 df=degrees_of_freedom,
 loc=sample_mean,
 scale=sample_standard_error)
 return ci

zapier_ci = calculate_ci(zapier_data['new_repo_count'])
ai_apis_ci = calculate_ci(ai_apis_data['new_repo_count'])

And just so I mentioned it, quick aggregate to compare Y/Y

def calculate_yoy_growth(data, year1, year2):
 jan_jul_year1 = data[(data.index.year == year1) & (data.index.month.isin(range(1, 8)))]['new_repo_count'].sum()
 jan_jul_year2 = data[(data.index.year == year2) & (data.index.month.isin(range(1, 8)))]['new_repo_count'].sum()
 return (jan_jul_year2 - jan_jul_year1) / jan_jul_year1 * 100

zapier_yoy = calculate_yoy_growth(zapier_data, 2023, 2024)
ai_apis_yoy = calculate_yoy_growth(ai_apis_data, 2023, 2024)

Plotting this result, it’s easy to see the divergence during the AI hype cycle timeframe.

# Create the plot
fig, ax1 = plt.subplots(figsize=(12, 7))

# Plot Zapier data on the left y-axis
ax1.plot(zapier_data.index, zapier_data['rolling_avg'], color='blue', label='Zapier')

# Set up the right y-axis for AI-related APIs
ax2 = ax1.twinx()
ax2.plot(ai_apis_data.index, ai_apis_data['rolling_avg'], color='red', label='AI-related APIs')

# Customize the plot
ax1.set_xlabel('Date')
ax1.set_ylabel('New Repo Count (Zapier)', color='blue')
ax2.set_ylabel('New Repo Count (AI-related APIs)', color='red')
ax1.tick_params(axis='y', labelcolor='blue')
ax2.tick_params(axis='y', labelcolor='red')

# Add legend
lines1, labels1 = ax1.get_legend_handles_labels()
lines2, labels2 = ax2.get_legend_handles_labels()
ax1.legend(lines1 + lines2, labels1 + labels2, loc='upper left')

# Set title and subtitle
plt.title("Public API Usage Trends Y/Y", fontsize=16, pad=20)
plt.figtext(0.7, 0.80, f"Zapier Y/Y Growth: {zapier_yoy:.1f}%, AI-related APIs Y/Y Growth: {ai_apis_yoy:.1f}%\n"
 f"(Based on Jan-Jul trends) * not statistically significant at 95% CI",
 fontsize=10, ha='center')

# Adjust layout
plt.tight_layout()
plt.subplots_adjust(top=0.85) # Adjust top margin to accommodate subtitle

# Show the plot
plt.show()

Does this correlate to any specific packages…? The plot below shows UIPath correlation — while this doesn’t equal causation, messaging from this company became aggressive in recent months towards the scholastic communities (free tools) — C3.ai data is dirty but also worth noting some correlation to Oracle AI and Google Vertex tools.

# Create a pivot table with months as index and keyword categories as columns
pivot_data = data.pivot_table(values='new_repo_count', index='month', columns='keyword_category', aggfunc='sum')

# Calculate correlation between Zapier and other categories
correlations = pivot_data.corrwith(pivot_data['zapier']).sort_values(ascending=False)

# Remove Zapier's self-correlation and any NaN values
correlations = correlations.drop('zapier').dropna()

# Get the top 5 correlated categories
top_5_correlations = correlations.head(5)

print("Top 5 dimensions correlated with Zapier:")
for category, correlation in top_5_correlations.items():
 print(f"{category}: {correlation:.4f}")

# Plot the correlation results for top 5
plt.figure(figsize=(12, 6))
top_5_correlations.plot(kind='bar')
plt.title("Top 5 Correlations (again, sans CI): Developer Usage of Zapier vs Other Categories")
plt.xlabel("Categories")
plt.ylabel("Correlation Coefficient")
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()

Synthesizing — what could this suggest?

1. Shift in Developer Focus in Past Year:

The declining trend for Zapier activity could indicate a shift in developer focus away from traditional automation platforms towards AI-centric technologies that were attempting to accomplish similar goals.

2. Recent Upturn for Zapier

The sharp increase in Zapier’s trend recently could be attributed to:

1. Introduction of AI-related Features: Zapier may have introduced new AI-centric capabilities or integrations, sparking renewed interest among developers.
2. AI hype may not have automated what developers were trying to do: There is no data to suggest this, since AI APIs are still increasing in usage.
3. Synergy with AI Technologies: The rise could reflect Zapier’s efforts to incorporate AI into its platform, possibly something involving free tools or UIPath, and also potentially offering new ways for developers to leverage both automation and AI capabilities together.

Caveats: It’s important to note that these trends may not capture the full complexity of the API ecosystem. Factors such as changes in Zapier’s business strategy, shifts in the broader tech landscape, and the emergence of new competitors could also play roles in shaping these trends (in theory).

Follow me for more insights on AI tool development and otherwise.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI