Topic Books and Reviews Aggregator: using C# and Google's Books and YouTube APIs

Working with APIs is really interesting, it is like building pipes to supply water in a city, in this case aggregation of information. This article will deal with writing a service that pulls book reviews for books based on a term I search for. In other words, I will input a term (one-word) that will be used to fetch up to 10 books on from Google Books then for each book will search Youtube for 5 videos each and aggregate them in a JSON file.

Will use Google Books V1 API and Youtube V1 API. Will be done using [C#] in .NET 5.

The source code for this is on GitHub: https://github.com/Morr0/Book-Reviews-Fetcher

The interface will be a console application and that’s sufficient for this case. You can always extend it for your needs.

Coding the basics:

Let’s start by making a new console project.

dotnet new console BookReview.App

Will now have a Program.cs. In the Main method will check that 2 arguments are provided, one of which is the topic to search and the other is the file path that I will save to.

if (args.Length == 0)
{
 Console.WriteLine("Please enter a topic (one word) as 1st argument");
 return;
}
else if (args.Length == 1)
{
 Console.WriteLine("Please enter a *.json to save to as 2nd argument");
 return;
}

string searchTerm = args[0];
string filePath = args[1];

Will create an appsettings.json and structure it as so:

{
 "GoogleApiKey": "kwjrfkwekf Api Key"
}

Next will install Microsoft.Extensions.Configuration.Json to configure the JSON configurations file to read from the API key.

dotnet add package Microsoft.Extensions.Configuration.Json

Now configuring the main method to read the API key. Note that you don’t have to follow this .NET standard (which is the same as ASP.NET Core), you could pass it as an argument or any other way.

var config = new ConfigurationBuilder()
 .SetBasePath(Directory.GetCurrentDirectory())
 .AddJsonFile("appsettings.Development.json", true)
 .AddJsonFile("appsettings.json")
 .Build();

string apiKey = config.GetSection("GoogleApiKey").Value;

Let’s temporarily printout in the console the API key.

Console.WriteLine(apiKey);

And run:

dotnet run Psychology results.json

Installing the Google clients:

dotnet add package Google.Apis.YouTube.v3
dotnet add package Google.Apis.Books.v1

Creating the Book abstraction:

Now I chose to breakout functionality into pieces following the Single Responsibility Principal. Will create a Abstractions directory.

Now I suggest you look at the following documentations to learn more about the search API of Google Books:

Although I will be using a client library (the official one). Knowing what is what is very helpful. Basically, a Volume is a book/magazine representation with information ranging from book, sales and many meta data you may find interesting. Since I will be using just some attributes of the returned information.

Will create the Book model class:

public class Book
{
 public string Title { get; set; }
 public string Subtitle { get; set; }
 public string Description { get; set; }
 public List<string> Authors { get; set; }
 public string Published { get; set; }
 public List<string> ISBNs { get; set; }
 public int PageCount { get; set; }
 public string PrintType { get; set; }
 public int Ratings { get; set; }
 public double AvgRating { get; set; }
 public string GoogleBooksUrl { get; set; }
 public double RetailPriceUSD { get; set; }
}

And will create the BooksAbstraction with a GetBooks method taking 2 arguments, an API key and a search query (which is the search term). Now will create the service:

public static List<Book> GetBooks(string apiKey, string searchQuery)
{
 using var booksService = new BooksService(new BaseClientService.Initializer
 {
 ApiKey = apiKey,
 ApplicationName = "BookReview"
 });
}

The way the Google client organizes/structures objects is quite different to AWS clients in .NET.

Will now create a ListRequest to describe what I want:

var listRequest = new VolumesResource.ListRequest(booksService, searchQuery);
listRequest.MaxResults = 10;
listRequest.ShowPreorders = false;

Here I am asking for a max of 10 results and not include any preorders which I am not interesed in.

Requesting, note there is an asynchronous version as well:

var listResponse = listRequest.Execute();

Will now make an list of Book and loop over all items in the list response and map each to a book model.

var books = new List<Book>();

foreach (var volume in volumes)
{
 var book = new Book();
 books.Add(book);

 book.Title = volume.VolumeInfo.Title;
 book.Subtitle = volume.VolumeInfo.Subtitle;
 book.Description = volume.VolumeInfo.Description;
 book.Authors = new List<string>(volume.VolumeInfo.Authors);
 book.Published = volume.VolumeInfo.PublishedDate;
 book.PageCount = volume.VolumeInfo.PageCount ?? -1;
 book.ISBNs = volume.VolumeInfo.IndustryIdentifiers.Select(x => x.Identifier).ToList();
 book.PrintType = volume.VolumeInfo.PrintType;
 book.Ratings = volume.VolumeInfo.RatingsCount ?? 0;
 book.AvgRating = volume.VolumeInfo.AverageRating ?? 0;
 book.GoogleBooksUrl = volume.VolumeInfo.PreviewLink;
 book.RetailPriceUSD = volume.SaleInfo?.RetailPrice?.Amount ?? -1;
}

return books;

Note that some properties maybe nullable so I assigned default values of -1 and 0. This is due to maybe some values non-existent.

Will also extract this last code snippet into a method taking the list of response items called MapResponsesToBooksModel return list of books. And return it’s content for the GetBooks method.

Now done on the books side.

Creating the Youtube abstraction:

I will be attempting the same abstraction for Youtube videos, however this is shorter, it will incur more API calls. I will search for 5 videos on each book title.

Will create the Video model class:

public class Video
{
 public string Title { get; set; }
 public string Url { get; set; }
}

Will call the abstraction VideosAbstraction. Will also add a constant of Youtube URL since the API doesn’t return the Youtube URL but only the Id of the video:

private static string YoutubeUrl = "https://www.youtube.com/watch?v=";

So the query parameter v is the id of the video.

The same way will create a GetVideos method taking API key and book title as arguments and returning list of videos:

using var youtubeService = new YouTubeService(new BaseClientService.Initializer
 {
 ApiKey = apiKey,
 ApplicationName = "BookReview"
 });

var searchRequest = youtubeService.Search.List("snippet");
searchRequest.Q = $"{bookTitle} Book Review";
searchRequest.MaxResults = 5;
searchRequest.Type = "video";

var searchResponse = searchRequest.Execute();

Note a very important thing, the snippet in the parameter of creating the search request, this is so I just pull the snippet data of information and not pull others. I suggest you look at the documentation to learn more:

Now will create the internal method that converts search response items:

private static List<Video> MapResponseToVideosModel(IList<SearchResult> results)
{
 var videos = new List<Video>();

 foreach (var result in results)
 {
 var video = new Video();
 videos.Add(video);

 video.Title = result.Snippet.Title;
 video.Url = $"{YoutubeUrl}{result.Id.VideoId}";
 }
 
 return videos;
}

Now just call the method and return it’s response models.

Calling and aggregating the response:

Now finished with the API abstractions, will create BookReviewResult which will contain an aggregate of book and videos for each book.

public class BookReviewResult
{
 public Book Book { get; set; }
 public List<Video> Videos { get; set; }
}

Back in the main method, will create a method to fetch books for the search term and for each book will call the Youtube abstraction to fetch videos for each book and aggregate them in a new instance of the model we made above.

private static List<BookReviewResult> GetBookReviews(string apiKey, string searchTerm)
{
 var bookReviews = new List<BookReviewResult>();
 var books = BooksAbstraction.GetBooks(apiKey, searchTerm);
 foreach (var book in books)
 {
 bookReviews.Add(new BookReviewResult
 {
 Book = book,
 Videos = VideosAbstraction.GetVideos(apiKey, book.Title)
 });
 }

 return bookReviews;
}

Calling it from Main method.

Writing to file:

This is quite simple, the following code will do. Pretty self-explanatory:

private static void WriteToFile(string file, List<BookReviewResult> results)
{
 string json = JsonSerializer.Serialize(results, new JsonSerializerOptions
 {
 WriteIndented = true
 });
 File.WriteAllText(file, json);
}

Now done with the application. Feel free to run it:

dotnet run Music info.json

Remarks:

Everything we’ve done here is my design choice. In the wild, similar types of services exist, where fetching data from multiple endpoints used as middleman APIs. This is typical of microservice architectures.

This type of work can also be done on batch jobs. Where a worker overnight/hourly will fetch hundreds/thousands/millions/whatever amount of data will consume database/APIs to arrange/aggregate data.

Note the Youtube API’s rate limiting, 100 points per search call. The quota can be found below: https://developers.google.com/youtube/v3/determine_quota_cost However, there doesn’t seem to be an announced limit on Google Books API. You can always submit for higher limits. I did not take into account errors due to going over limit.

Conclusion:

Riding through API calls to an end product is a really interesting process. This is the basis of how inter-application communication works.

Thanks for reading this.