Operations on RecipeRadar Blog

Maintenance: upcoming RecipeRadar server migration

Wed, 01 Jul 2026 00:00:00 +0000

The RecipeRadar website and recipe search APIs are currently hosted on a single home computer system, previously used a general-purpose desktop workstation and gaming computer.

That computer will soon need to be powered-down, potentially for a duration of weeks.

So the question arises: how can we keep the RecipeRadar application running during that time window?

The answer is that we are renting some temporary Virtual Private Server hosting from 123-reg.co.uk, the same company that manages the DNS records for RecipeRadar.

As a user of RecipeRadar: you should not notice any functional changes in the RecipeRadar application during or after this migration.

Because the server we are renting will have different computing resources (CPU speed, memory, network) and will be located elsewhere, you may, however, notice some differences in speed/performance of RecipeRadar.

In addition, the IP address and TLS (aka SSL, HTTPS) certificate for RecipeRadar will change.

The updated IP address for RecipeRadar will be: 37.148.205.122

The DNS B record – the webintegrity record for the RecipeRadar application that runs on your device when you use RecipeRadar – will not change during the migration, although it may be updated again when future application updates are applied.

For more details and to follow progress on the tasks that take place during the server migration, visit: https://codeberg.org/openculinary/infrastructure/issues/69

WebIntegrity: a prototype for DNS-based website integrity

Sat, 17 Aug 2024 00:00:00 +0000

RecipeRadar attempts to provide a transparent, reliable and trustworthy recipe search experience. On this blog, we’ve previously written about a key aspect of that: our principle of using Open Source software whenever possible.

However, declaring that we use open source software does not, in itself, provide assurance of service integrity. In particular: transparent and reassuring documentation about a named network communication service is a necessary prerequisite for trust, but users should also confirm that that they are indeed connected to that same named service – and not some other intermediatry.

In information security, a classic disruptive pattern that aims to subvert even the most honest of communications is the Adversary-in-the-Middle attack. When we send a letter to a friend, we would like to assume that they (and only they) receive it. However, were someone else be able to intercept that message, then, in the absence of other protective measures, that intermediate person could create a subtly-adjusted forgery, and perhaps send that on to the intended recipient.

There are various ways to guard against this kind of disruption. Two powerful techniques include: agreement of pre-shared authentication measures – ways for each partner to confirm that their messages are genuinely from each other, and that an adversary is unlikely to be aware of – and the use of multiple channels of communication, so that an adversary would find it more difficult to intercept all of the relevant communications, something that they might need to do to keep their fabricated story believable.

RecipeRadar provides service as part of the World Wide Web, one of the most (if not the most) open and democratic network environments that we are aware of. On the web, service provision typically involves two key technologies: the Domain Name Service (DNS) protocol – operated by Internet infrastructure providers – and the HyperText Transfer Protocol (HTTP) – operated by individual website owners.

Commonly-received wisdom on the web states that using Transport Layer Security (TLS) - sometimes anachronistically referred to as Secure Sockets Layer (SSL) - can provide integrity. In brief: TLS is the feature that distinguishes plain-text HTTP communication from encrypted HTTPS communication: on the public web, TLS is implemented using public-key cryptography in combination with strict governance of the Certificate Authorities (CAs) who are able to mint trusted certificates.

It is true that TLS does provide integrity: if you are connected to a web service that uses TLS, then the messages sent to-and-fro can leverage the cryptographic properties of the communication channel to confirm, to a very high degree of probability, that messages have not been tampered with. However, typically only the pair of communicating peers are able to observe those communications, and as a result, the recourse available to a client communicating with an individual server that has obtained a valid certificate through dubious means is limited.

Is there a way that we could achieve integrity for web users without relying upon secrecy? We believe that there is - and we bootstrap our approach using an existing W3C standard called SubResource Integrity (SRI).

SubResource Integrity allows a web browser – an HTTP client – to check whether the files (resources) mentioned on a webpage have the expected content if the browser chooses to download them. To do this, the webpage must mention not only the filename, but it must also provide an integrity attribute that contains a checksum – or multiple checksums – corresponding to the expected content.

So, when your browser downloads the index.html page for RecipeRadar, we include the integrity HTML attribute on various subresources referenced in the page. So far, so good; but how can you trust that you have received the intended index.html page?

Our solution is naive and limited, but is backwards and sideways compatible: we re-use the checksum format specified by W3C SRI, and place it into a DNS B TXT record. At the time of writing, RecipeRadar returns the following responses from a DNS TXT query:

$ dig -t TXT reciperadar.com

; <<>> DiG 9.20.0-Debian <<>> -t TXT reciperadar.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29630
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;reciperadar.com.               IN      TXT

;; ANSWER SECTION:
reciperadar.com.        3600    IN      TXT     "v=spf1 include:_spf.google.com ~all"
reciperadar.com.        3600    IN      TXT     "B=sha512-zq+vAQPEmB9RpcNiexq0+GLJqGsWim5jZskS1OtdnohK15D+UbQxGmKEeHQgEDiX1bK4c3CdygHdRlOlLEsGEQ== sha512-eRMiiXuO5gHP0byDCCxlc3KqA29H7SE+6y20/73cYEIiv7/m9ottwYm5GSbrjL6hhH9+JBPbD6s7a0rk3ZmSHA=="

;; AUTHORITY SECTION:
reciperadar.com.        3600    IN      NS      ns29.domaincontrol.com.
reciperadar.com.        3600    IN      NS      ns30.domaincontrol.com.

;; Query time: 36 msec
;; SERVER: 192.168.0.1#53(192.168.0.1) (UDP)
;; WHEN: Fri Aug 16 23:05:36 BST 2024
;; MSG SIZE  rcvd: 350

This communicates that an HTTP GET / – a web request to the root path of an HTTP server – to the reciperadar.com domain – is expected to return content that matches one of the two SHA512 hash values included in the B record. Any other response content should be considered problematic by the HTTP client.

Publishing more than one active checksum means that we can deploy an updated version of our static web application while simultaneously allowing for the existing/stale version to remain valid. When we become confident that clients reading current DNS results should receive fresh HTTP content, we can remove the outdated checksum.

Niche Software Libraries: Volume I

Tue, 19 Sep 2023 12:53:54 +0100

When it’s going well, software development can be as enjoyable and straightforward as building a model from LEGO bricks: you have an idea or a goal in mind, and you begin to scour through the available components to put together the intended result. If you’re lucky, you find detailed instructions – requirements and documentation – to help you along your way.

Meanwhile, one of the reasons that open source software has become so successful over the past few decades is that different software developers are frequently looking for components that match similar requirements when they’re building systems. Regardless of the size of a software project or the industry that it relates to, common challenges appear again and again during development: parsing text, storing structured data, monitoring system uptime.. the list is endless. And thanks to the Internet, once a previously-unsolved problem has been addressed, it’s straightforward to share and distribute the code that filled the gap.

Over time, this creates a form of self-improving software community workshop where developers can demonstrate that their components are the most useful, highest quality, best tested, well maintained and easiest to use – and then selectively retrofit and improve their own workflows and projects based on each other’s contributions. The rewards for participating are numerous: there’s a feelgood factor when authors learn that they’ve enabled someone else to achieve something, there’s a name recognition factor, there’s the possibility to discover career opportunities, there’s social interaction, and there’s the chance to share the development workload.. not to mention the satisfaction gained from using the software as it improves, or the potential to obtain funding to develop it.

The resulting software commons - which has no single authority or geography - has produced widely publicised projects such as the Linux kernel, GNU Coreutils, and PostgreSQL database that have undoubtedly altered the course of computing history - and in doing so, have made software easier, cheaper and less time-consuming to access, deploy, learn, and improve upon.

This post is about some of the smaller, lesser-known libraries that RecipeRadar has investigated and selected for specific functionality so far. As a reminder: our objective is to provide completely free recipe search and meal planning to the general public, and to build that functionality in the open, using free and open source software (FOSS).

peer-base and yjs

Today, RecipeRadar exists as a single-user application: you open it on your computer, tablet or smartphone, and although your device communicates with the RecipeRadar API, your copy of the app does not communicate with anyone else’s.

In the past we’ve experimented with introducing peer-to-peer networking to enable scalable and fault-tolerant multi-user interaction: a shared app session could exist across two or more smartphones, perhaps even during outages of the RecipeRadar search service. The initial use case for that was to enable multi-person households to co-ordinate food shopping; a companion task to meal planning.

One of the core challenges in a peer-to-peer user application is how to maintain consistent state between each client – in other words, for each user to have the same model of the world while they use the application. This is particularly tricky because user devices aren’t online continuously. Your housemate might turn on airplane mode to save battery while doing the weekly shopping, and you might be editing the shopping list from home at the same time. What happens to the combined edits from both of you when your housemate re-activates their data connection?

The job of a good technology library is to hide (or to ‘abstract’, to use industry terminology) as many of the details of questions like these from developers. Writing a library that exposes a sensible set of operations that makes it easy and straightforward to achieve objectives while making it difficult or impossible to misuse the system is a large part of good API design.

Among libraries that provide support for multi-party data consistency, there are two popular modern schools of design: Operational Transformations have proven popular in some cloud-based collaboration suites, and Conflict-free Replicated DataTypes provide an alternative that does not require a central co-ordination authority - arguably a benefit, but also a drawback under some circumstances, because the ability to decide what edits are valid can be difficult in a distributed environment, and especially so in situations where some clients may be untrusted or have software bugs (that is: reality).

We decided to explore the edge of what’s technically feasible, and began exploring CRDTs. Of the available CRDT libraries compatible with the ecosystem of our application (JavaScript within a web browser context), two libraries in particular appeared promising to us:

peer-base - a library to assist the construction of distributed peer-to-peer applications; there isn’t much evidence of developer activity in the past few years although there is at least one impressive video demonstration of an app that uses the library: peer-chess.
yjs - an actively-maintained and fairly popular CRDT-based library that provides integrations with some existing text editing components, including ProseMirror.

We also learned about Cattaz, a research project by Fujitsu Labs that builds upon YJS to demonstrate rich collaborative editing capabilities as part of a wiki-based document editing tool.

RecipeRadar is registered as a company in the United Kingdom, and although it may be an overly-cautious interpretation of the draft Online Safety Bill regulations, we determined that allowing our users to communicate with each other, even within a potentially limited scope such as shopping list contents, could open us up to additional technical compliance requirements. Such functionality may still prove worthwhile in future, but we’d like to wait to see how the regulatory landscape evolves before spending development effort in this area.

recipe-scrapers

Retrieving recipes from the world wide web and making them searchable is the core of what RecipeRadar does, and the recipe-scrapers library is a key component that simplifies that process for us: it is a library that accepts the HTML of a recipe webpage as input, and it provides access to discrete recipe-related fields (the title of the recipe, the author’s name, the list of ingredients, and so on) as output.

Where possible, recipe-scrapers reads recipe information from a standard schema.org format that many recipe websites have adopted - and in situations where that isn’t possible, it can attempt to find the same data using another library, beautifulsoup that is purpose-designed to query the contents of HTML and XML documents.

We chose to integrate recipe-scrapers partly because the programming language that it’s written in, Python, is the one that we’re most familiar with and that we use throughout most of the the RecipeRadar technology stack.

recipe-scrapers has a friendly, collaborative developer community that we participate in, and we try to walk a careful line of encouraging and improving the functionality that the library provides while not pulling it in a direction that favours RecipeRadar to the detriment of others. Generally we expect that the use cases we have for the library are similar to those of other people who use it – that is to say, we think that our incentives are broadly aligned with those of its’ community, and we consider it important to maintain that alignment.

It doesn’t do everything that we need; recipe-scrapers won’t extract ingredient names, quantities and units from each line in an ingredient list, for example. But in software, that can be a good thing: each component should focus on what it does well, and subtasks can be delegated. Roughly speaking this is referred to as the principle of separation of concerns in software engineering.

hashedindex

Search engine technology is important throughout RecipeRadar - not only to provide the recipe search functionality that we make accessible through our web application, but also during processing of recipe text.

Part of our work involves maintaining a dataset of more than two thousand named products (ingredients). We use that dataset as a reference when we crawl recipes from the web, and we attempt to annotate each ingredient listing with a relevant product name.

Matching that text is itself a search problem: what is the best product match within a snippet of text such as block of cheddar, finely grated?

To produce the answer to that question (cheddar), we build a search index of known product names, and then we use the input text – the ingredient’s description – as a query to find and rank the results. That might seem slightly counter-intuitive: we’re using content from the web as a query, and searching across a set of documents that are typically very short - cheddar is likely the entire content of the document we’re looking for in the given example. This contrasts to most search engine use cases, where queries are usually shorter than the documents that are retrieved – but in this case, the situation is reversed.

Why did we use search engine technology to implement this solution? Well, it’s largely because it’s an area of technology designed to deal with the kind of linguistic ambiguity that occurs in text that was written by and intended for humans, where spelling mistakes and differing word endings (plurals, for example) can occur, and where ranking of multiple potentially-relevant results can be important.

The hashedindex library provides a neat Python-based implementation of text tokenization and search indexing, providing the basis for our in-process Python product search technology.

Thanks!

That’s all for this post! Thanks for reading - we’ll share what we’ve learned about more of the open source libraries used within RecipeRadar on this blog in future.

Ingredient parsing: handling plurality

Thu, 25 Feb 2021 18:01:12 +0000

This is the second post in a series that explores the technology that RecipeRadar uses to process and store recipe ingredients

In part one, we introduced a simplified version of the data model and XML format that RecipeRadar uses to represent ingredient information.

In this post, we’ll explain how we handle singular and plural ingredient names for English-language ingredient text.

Let’s begin by presenting two of RecipeRadar’s ingredient search use cases.

Use case 1: Search-by-ingredients

RecipeRadar should allow users to search based on the ingredients they have available, without having to specify quantities. A user may have tofu and a red onion available at home, for example, and may want to exclude recipes that contain beef.

In English, the words ’tofu’ and ‘beef’ do not have plural forms, so we will focus on the ‘red onion’ ingredient.

For this user’s query, we’d like our search results to include recipes that contain either red onion (singular) or red onions (plural) as ingredients.

Use case 2: Ingredient autocompletion

RecipeRadar should provide relevant autocompletion suggestions when the user begins entering an ingredient name.

For example, if the user has entered the characters green ol, then we may want to display either green olive or green olives as a suggestion, but we do not want to display both – they are the same ingredient.

Design Considerations

As with many system design problems, there is more than one possible solution, and trade-offs are involved when choosing an approach.

Some of the questions that we asked ourselves were:

How can we ensure that a query for an ingredient will match recipes that contain it in either singular or plural form?
How can we determine whether to suggest the singular or plural form of an ingredient during autocompletion?
How can we minimize application complexity?
How can we ensure that search responses are returned quickly?

Chosen Approach

We currently use the following strategy:

Store the original plurality for each ingredient named in the source recipe
Also store the singular form of each ingredient name
Require that searches use the singular form of the ingredient name
To determine suggestion plurality, count the number of times singular vs plural was used, and display the more ‘popular’ form

This approach is relatively wasteful in terms of storage, since we store both the original and singular forms of each ingredient.

However, it is also relatively simple to explain and implement, and can be achieved without managing a set of ‘preferred pluralities’ for each ingredient.

It also has a nice property that it will automatically adjust to reflect content retrieved from the web. For example, if most recipes switch from using a traditionally singular ingredient name to a plural form in future, then RecipeRadar’s suggestions will adapt to that over time.

Implementation Reference

At the time this article was written, most of the relevant code is found in these places:

Storage: the product name and related fields in the RecipeRadar ingredient model
Storage: the ingredient product properties stored in the Elasticsearch Mapping
Autosuggest: the plurality detection code, which uses Elasticsearch Aggregations
User interface: the autosuggest result processing code, which uses the select2 library to place each ingredient’s singular name into input fields, where they are later retrieved when a search is performed

Keeping busy with maintenance and bugfixes

Fri, 12 Feb 2021 00:00:00 +0000

The team learned of some existing issues in the black Python code formatter that we use to format some of our published code, and is providing time to help.

The team has also been working on maintenance and bugfixes since the last fortnightly update:

Preparations have been put in place to update RecipeRadar when the release of SQLAlchemy v1.4 lands, which is anticipated within the next few weeks
A serious user-facing bug that prevent recipe scaling was identified and fixed - we believe that this affected users who retrieved the application between 2021-01-06 and 2021-02-08
We have migrated handling of fractional amounts in the application to use a purpose-designed library, fraction.js
An updated version of hashedixsearch was released
Progress on collaborative editing has continued, and most recently we’ve begun experimenting with Markdown syntax highlighting of shared text

Coming up next

There are a few different directions we would like to continue in:

Ensuring that the credits on the site are updated to reflect more of the contributions included so far
Providing NPM packaging for our published feedback.js modifications could be beneficial long-term, even if we are the only consumers of it for now
Multi-user and multi-device collaboration, as ever, remains a product area that we would like to progress
Continuing our blog series on ingredient parsing would be worthwhile

Video demonstration: multi-device collaboration!

Fri, 08 Jan 2021 00:00:00 +0000

This week’s update includes a video feature-preview of the (oft-touted, rarely progressed) multi-device collaboration feature.

The user interface is intended to be as straightforward as possible. You’ll pick a username and a ‘room name’ - a shared name for the place you’d like your meal plan and shopping list to be available - and by providing that room name to your friends or other devices, they’ll be able to participate equally in the session.

There’ll be no user account to create - and thus no login details to remember or worry about, no unwanted marketing or risk of spam emails - and none of your personal information will be shared with RecipeRadar.

Attached is a short video clip of the feature as it currently exists in development - demonstrating me adding a note to a shared shopping list from another device.

As usual, we’re building upon functionality provided by other free and open source components. In this case credit is due to the YJS collaborative editing library, and the CodeMirror text editor.

Reflecting on 2020

Fri, 18 Dec 2020 00:00:00 +0000

Nutritional Information

The main item of news today is that we’ve launched display of nutritional information in the application. Not all – and in fact, not many – recipes that we’ve collected so far have nutritional information included alongside them, but when present that information will now be displayed in the search results.

We’re currently refreshing (or ‘reindexing’ as we call it within the team) the recipe data on RecipeRadar, and more nutritional information will appear as this process continues.

Reflection on 2020

Rather than provide a lengthy update this week, we’re going to reflect on some of the features added during the year.

During 2020 we’ve added:

Smarter ‘partial’ name matching during ingredient search
Recipe yield scaling
A recipe detail page that can be shared via hyperlink
Dietary properties (vegetarian, dairy-free, …) in recipe search results
Filtering of search results by recipe website
Tolerance for spelling errors during ingredient search
A recipe ’explore’ view that provides simple yes/no ingredient prompts
…and last but not least, rendering of nutritional information

We were a touch optimistic on some of our planned roadmap, but that’s ok; work will recommence fresh next year, and perhaps we’ll be better prepared to plan future tasks with the added experience of hindsight.

A new way to find recipes: the Recipe Explorer!

Fri, 04 Dec 2020 00:00:00 +0000

During the last two weeks, we had a relatively radical idea for a way to explore recipes on RecipeRadar.

Rather than prompting people to enter the ingredients they have (requiring a modest but noticeable amount of data entry), we can present a list of common recipe ingredients to the user, and allow them to use swipe gestures to say yes/no to the ingredients that they have, saving on typing.

As they swipe to indicate whether each ingredient is available, the list of subsequent ingredient choices updates dynamically based on the recipes that match the ingredient selections made so far.

A beta version of this interface is now live - please try it out if you have time, and send us some feedback!

Here’s a screenshot to give you an idea how this works:

Coming up next

Work continues towards restoring multi-user collaboration (as we seem to keep on harping on about in these updates). It’d be great to re-release that functionality before the end of the year; let’s see how we get on.

We are also planning on generating a web of static content pages based on the recipe and ingredient metadata we have within RecipeRadar.

The goal is for these pages to be genuinely useful to users who are looking for information about ingredients while also providing a search-engine-friendly content directory that generalized search engines like Google, Bing and Yahoo can index.

Blog updates, filter-recipes-by-website, and more

Fri, 20 Nov 2020 00:00:00 +0000

Welcome to another almost-fortnightly RecipeRadar update.

There have been a few developments since the last update:

Our first feature-related blog post, “An introduction to ingredient parsing” went live in early November - part of a series with follow-up posts coming soon
We’ve completed a large-scale refactoring (re-organization of code and data) to make it easier to manage and correct the information that we store about ingredients. As a side-effect, this has further improved our recipe indexing performance.
We’ve added the ability to filter search results by recipe website. If you have a favourite recipe website, or if there are websites that you don’t like, you can now choose to show/hide those in your search results.
The application now displays dietary properties alongside each recipe, indicating whether they are dairy-free, gluten-free, vegan and vegetarian

Coming up next

The largest work-in-progress feature is the long-awaited collaboration feature, restoring multi-device application sessions.

It’s proving difficult for the engineering team to get traction on - the design in particular is fairly complicated and important to get right, but we’re working on it.

Additionally, we have some smaller features that we’re hoping to look at soon:

Adding a ‘share link’ option from the recipe view page may be a quick win
We may be able to improve search performance by removing some unused fields

Finally, we’re still hoping to surface recipe nutritional information once this is available via the recipe-scrapers library. More news on that when we have it.

An introduction to ingredient parsing

Fri, 06 Nov 2020 11:02:32 +0000

This is the first post in a series that explores the technology that RecipeRadar uses to process and store recipe ingredients

As a recipe search engine, RecipeRadar aims to collect and understand recipe content from around the web.

We’re able to crawl recipes from any website supported by the recipe-scrapers Python library, and the results of each crawl include – among other details – a list of the recipe’s ingredients and a list of the preparation step involved. Both of these lists are represented as lines of text, as written by the original recipe author.

For each ingredient there are two key pieces of information we’d like to identify: the name of the ingredient, and the associated quantity of it that is required. Using these, we can provide search-by-ingredient functionality – “show me recipes containing tomatoes” – and also turn a meal plan into a shopping list that includes sum totals of the ingredients required.

Let’s use some real-world ingredient descriptions as examples. One recipe could contain an ingredient with the description “2 cups of vegetable stock”, while a second recipe may list “30ml vegetable stock”.

Within RecipeRadar’s data model, we refer to the purchasable contents of an ingredient as the ‘product’ (vegetable stock in both of these examples), and the amount required as the ‘quantity’ (2 cups and 30ml respectively).

So far, so good. The situation becomes a little more complicated when we realize that recipe authors add all kinds of useful flair and detail to their ingredient descriptions:

a pinch of freshly ground black pepper
2 large apples, cut into chunks
a bunch of kale, roughly chopped
one medium apple

Here we find products (black pepper, apples, kale) and quantities interleaved with additional descriptive details (freshly ground, …). We’d like to keep that extra information around – it can help with preparation and selection of ingredients, and it helps to ensure that the recipe author’s instructions are faithfully maintained.

Although inter-service communication within RecipeRadar typically uses JSON to serialize objects and data, we’ve found that JSON is not well suited to representing text with inline metadata. Instead, we use XML which allows us to add markup that is attached to individual phrases.

Taking the first example from the ingredient list above, we could use XML to annotate the product and quantities like so: a <quantity>pinch</quantity> of freshly ground <product>black pepper</product>. In practice, RecipeRadar uses a slightly different XML schema but the principle is the same.

In our next blog post, we’ll cover some of the code that extracts product names and ingredient quantities from plain text, and explain how we use the resulting information to provide recipe search and meal planning functionality.

Construction continues.. and an update on indexing performance

Fri, 23 Oct 2020 00:00:00 +0000

At RecipeRadar engineering HQ, feature development has been continuing - at a slightly unpredictable pace - in a few parallel areas.

Collaboration (unblocked)

As mentioned during the previous update, we had been blocked on multi-device collaboration work due to an issue affecting database unit tests. The good news is that the issue has since been fixed!

Work on multi-device sessions should recommence shortly.

Surfacing nutritional information (blocked)

Display of nutritional information within the application is in progress, but we’ve hit some snags estimating aggregate nutritional information for recipes.

In short, there are too many gaps in the data for us to accurately predict recipe nutrition from the ingredients where we have known metric quantities.

We’re likely to wait until nutritional information is supported natively in the recipe-scrapers library before continuing. We’re optimistic that this feature will simultaneously allow us to display accurate nutritional information, while also allowing us to backfill gaps in our knowledge base.

Dietary properties (landing soon)

Support for recipe dietary properties (‘dairy free’, ‘vegetarian’, …) is in progress, and should be ready for user testing next week.

The RecipeRadar application will display icons to indicate the dietary properties of recipes in the search results, and also allow searching/filtering by those properties.

(Re)Indexing performance

Recent work related to recipe metadata has highlighted performance problems during (re)indexing of search engine contents.

Currently it takes approximately two days to reindex the entire recipe document set, which makes development iteration challenging.

We’ve spent time to identify areas for performance improvement and it appears that disk storage is our current bottleneck.

We may need to order some new disks for our server to make significant improvements; this work is planned and tracked in infrastructure issue #27.

Coming up next

Resuming work on collaborative sessions and surfacing additional recipe metadata in the application will be our primary focuses over the next two weeks.

The past few weeks have been very consciously focused on data and enabling features, and we hope to revisit application UI and UX improvements in the near future.

Multi-device collaboration, and a discussion on Hacker News

Fri, 09 Oct 2020 00:00:00 +0000

Well, it’s been a little over a fortnight since the last update; despite that, progress continues on RecipeRadar.

Multi-device Collaboration

After some deep thought regarding a technical design for multi-user collaboration on RecipeRadar, we now have a specification written up. The engineering team is working towards implementing this.

Most good software includes tests to ensure that the code does what it intends to and that it doesn’t break during future changes. Unfortunately we’re hit a slight roadblock when attempting to write unit tests that cover the database functionality included in the application. The team has reported the issue upstream and is working on a possible fix.

Hacker News Discussion

There was a recent discussion thread on the popular ‘Hacker News’ technology forum regarding recipe search websites. You can read the entire thread here; RecipeRadar participated in the conversation, and as a result:

We fixed some issues related to missing tuna-and-teriyaki-related recipes
We received some feedback and ideas regarding ‘staple’ kitchen ingredients
Some users contributed suggestions regarding kitchen pantry inventory management integrations

Coming up next

We’re hopeful that work on multi-user collaboration will soon become unblocked.

While we’re waiting for that to be the case, there are also a few smaller feature items that may we start work on, including showing some of the ingredient nutritional data (that we recently integrated into the knowledge graph) alongside each recipe.

Progress update on recipe metadata extraction

Fri, 28 Aug 2020 00:00:00 +0000

This fortnight’s update is relatively short; the engineering team is getting back up to speed following some holiday/vacation time.

Nutritional Information

Development towards including ingredient nutritional metadata is making good progress - the team’s been in contact with the author of the open source ingreedy-js project that has already constructed an excellent dataset of nutritional metadata. That dataset has now been fuzzy-matched and included into the RecipeRadar ingredients knowledge base.

The next step will be to index that metadata into our search engine so that approximate aggregate nutritional data is available to surface in future application features.

Direction Metadata

It looks like Facebook’s duckling natural language parsing project could be very helpful to extract additional information – temperatures, for example – from recipe directions. We would prefer to run this tool as a service within our Kubernetes container environment, and in order to do so we’ve proposed a small fix/change upstream to them and are awaiting feedback.

Coming up next

These tasks are likely to continue within the next two weeks, and after that we’ll be working on one of our most important long-term pieces of functionality: collaborative (aka ‘multiplayer’) RecipeRadar sessions.

Getting started with Nutritional Information

Fri, 31 Jul 2020 00:00:00 +0000

While there’s no single large feature that has landed in RecipeRadar in the past fortnight, there are a number of irons in the fire.

This week’s update is primarily textual rather than visual – we don’t have any screenshots to share this time – but as you’ll see, that’s appropriate given some of the work we’ve been doing.

The items we’re making progress on currently include:

Nutritional Information

We’ve identified a number of trustworthy and authoritative sources of ingredient nutritional information, including the USDA’s FoodData Central and UK’s CoFID dataset.

The next challenge is to find a good way to associate the records from these datasets with the ingredient identifiers that RecipeRadar uses internally, and to update the knowledge-graph so that it can store and serve the associated metadata.

Please note that we don’t intend to surface this nutritional information to users via the application during Q3; we do however have plans to offer functionality related to dietary planning in Q4, and that will incorporate nutritional factors.

Direction Metadata

Over the past couple of weeks, we’ve explored various options related to extraction of metadata from recipe text – we want to identify the various actions involved when preparing recipes ("combine the flour and milk") and also to discover the important entities involved in each step (“pre-heat the oven to 200 F”).

The ideal outcome of this – which could take some time – is that we will be able to use the co-references discovered in recipe directions to represent those visually, as in the example diagram included in the application’s vision statement.

As part of the work towards this, we have re-structured some of the responses of the knowledge-graph so that they return data in XML format. The primary driving reasons for this are that XML can represent data as a tree/graph (which we want) and also that it can represent entities interleaved in-line with natural language text (which we also want).

In addition, we are now performing basic identification of verbs in recipe directions. This is relatively straightforward thanks to modern open-source natural language processing libraries. We’ll need to introduce more advanced techniques from these libraries in order to reach a better understanding and representation of recipe text.

Coming up next

It’s hard to say precisely where we’ll get to with each of these goals within the next two weeks – perhaps moreso because our entire engineering team (currently a euphemism for ‘me’) will be on holiday until mid-late August.

Integrating nutritional data before then seems like an ambitious but achievable goal, and we may send a mini-update if we manage that.

Either way, you can expect the next ‘fortnightly’ update roughly a month from now.

Internal tooling: recipe crawler diagnostics

Fri, 17 Jul 2020 00:00:00 +0000

The last couple of weeks have been slow and steady in terms of RecipeRadar software engineering progress.

The main focus recently has been on providing recipe data debug tooling in order to help identify and fix future data quality issues. Here’s an overview of how that works.

Recipe Data Diagnostics

When users are searching for recipes, it’s important that we display accurate representations of the ingredients and directions for each recipe.

RecipeRadar takes recipes from all over the web and translates them into a common representation in a format called JSON, with a lot of help from the ever-improving recipe-scrapers library.

Sometimes, the contents or format of the recipe website may change – and sometimes our own processing of the data may change (for example, after we learn about a new ingredient or item of kitchen equipment, we can add additional metadata to the JSON format).

The newly-added recipe diagnostics page shows the contents of the recipe, as it was last indexed into RecipeRadar’s search engine, and also as it appears currently when applying the latest crawling & post-processing operations.

How does that all look in practice? Well, here’s an example:

In this case we can see that the version of the recipe in the search engine contains 19 unique ingredient products (note the strikeout text in red), and the newly-crawled version of the recipe contains 21 – one new ingredient is visible in the large block of green text.

That means we should probably re-crawl and re-index this recipe so that users can find the updated (and in this case, more accurate) representation of the recipe.

Something’s still a little unusual in this case however - the singular form of the product name is ‘red chilly’. It’d be worth investigating and resolving that as well.

Coming up next

It’s uncertain exactly what feature we’ll be working on next, but it’s likely either going to be gathering ingredient nutritional data, or it’ll be figuring out how to support ingredient substitutions (for example, substitute spices or proteins).

Spelling suggestions are live!

Fri, 03 Jul 2020 00:00:00 +0000

Q2 2020 is now complete, and I’m glad to say that we managed to achieve almost all of the goals for the quarter. The only task that has rolled over into Q3 is the selection of a graph database technology, in order to support the knowledge-graph component.

Spelling Suggestions

During the last update I mentioned that we had discovered the fast-autocomplete software library as a candidate to provide spelling suggestions on RecipeRadar.

Although that project does look great, we discovered an issue with it during integration. Out of the box, it doesn’t provide search suggestions when you search for a word that is in the middle or at the end of an ingredient name.

For example, searching for ‘cream’ would not suggest ‘sour cream’ using the default fast-autocomplete configuration.

That’s significant because it’s an existing feature that we support and would like to retain for RecipeRadar - it’s useful and has been requested in the past.

As a result, we’ve decided to re-use our existing Elasticsearch search engine to provide ingredient spelling suggestions.

When you start entering some text, the application looks in the set of ingredients for near-matches on any word in the ingredient:

I’m glad to report that this feature is now live on RecipeRadar; please give it a try and let me know what you think.

Coming up next

With the previous quarter recently complete, it’s a good time to look further into the horizon to see what’s planned over the next few months.

Among the most promising planned features are:

Restoring multi-user collaboration support, so that you can share your meal planning session with friends and family
Providing ingredient substitutions, so that you’re able to find more recipes that use the ingredients you have in stock
Gathering nutritional information into the knowledge graph, so that we can use that information to unobtrusively guide users towards making healthy eating decisions

These are ambitious, and will run in parallel with our regular work including bugfixes, incrementally adding user interface improvements, and recipe search result tuning.

We may have to rearrange the roadmap tasks a little as we learn more about the size of work involved for each feature, but this is what we’re setting out to do, starting on Monday.

Introducing the Recipe Detail Page

Fri, 19 Jun 2020 00:00:00 +0000

Recipe Detail Page

To make better use of device screen space, clicking on a recipe now takes you to a full-screen ‘recipe detail page’ where everything needed for meal preparation is displayed on a single bookmarkable page.

The page incorporates a number of aspects from Monica’s screen designs, including the ability for the user to hover-to-highlight recipe direction steps.

Recipe Scaling

If you’re eagle-eyed (or had been waiting for this feature) then you may have spotted that the ‘servings’ field on the recipe details page is an edit box.

When viewing a recipe (via the recipe detail page) you can now adjust the number of servings, and the quantity of the ingredients will automatically be updated to match.

Note that this feature isn’t hugely ‘smart’ yet. It doesn’t, for example, understand that the concept of one-and-a-half eggs may not make much sense. There’s more work planned to improve the functionality it provides - and it would benefit from your feedback and testing, too!

Coming up next

We’ve made some progress investigating how to provide spelling corrections during ingredient name search on RecipeRadar, and have discovered a open source software library called fast-autocomplete that looks ideal for the purpose.

Work has begun on integrating this library with RecipeRadar, and there should be more news on this soon.

We should also be making some decisions regarding selection of a user interface framework (for the application interface) and a graph database (for ingredient knowledge representation and storage) within the next two weeks.

Database architecture updates and an internationalization question

Fri, 05 Jun 2020 00:00:00 +0000

Updated Website Prototypes

One of the features of the screens that may involve the most complex user interactions and information presentation is the selection, display and navigation of planned meals (calendarization).

Architecture Investments

When you interact with RecipeRadar on your phone or on your computer, the information presented to you is served by our recipe search engine, which is based on the open source Elasticsearch product.

We also use the popular open source PostgreSQL database to store persistent data: ingredients, recipes, our progress crawling the recipe web, and user search logs (which do not contain any personally-identifying information).

Until recently, both the search engine and database have had to be online continuously for the application to work correctly, and upgrades to the latter were performed manually.

The main risk introduced during manual database upgrades is that there are fewer guarantees that the changes will be applied and will behave the same way that they did during development.

I’m glad to say that following some modifications, the entire RecipeRadar application experience can now be served by the search engine API, even when the database is temporarily unavailable. Support has also been added – albeit not yet thoroughly tested – for scripted database upgrades.

These improvements reflect progress against our tentative product roadmap.

Company Accounts

Although I don’t have much feedback to report yet, I’ve been spending a little time investigating potential options around company financial reporting and accounting. This will take place in the UK and I hope that there will be a good level of transparency achieved in the process.

Internationalization Support

Does anyone on our list have experience using Weblate?

It looks perfect for our translation use cases; it’s open source, and provides ‘continuous localization’ – the idea is that translators’ work should be incorporated into the application on a regular basis, matching the cadence of application releases.

Ideally, if we release two updates for RecipeRadar during a given day, and a contributor provides a verified, approved translation in the time between those two releases, then it should be included in the second release).

Coming up next

Looking at our Q2 2020 roadmap, we have a few user-facing features that we’re aiming to implement over the next few weeks. The next items we’ll be looking at include:

A ‘recipe detail’ page in the application
The ability to scale up/down the number of servings for recipes on the detail page
Support for spelling corrections during ingredient and equipment search

It may be a stretch to achieve all of these within a two-week duration, but we’ll see how we get on. Expect to hear some updates on these in the next fortnightly update.

Partial ingredient search, meal planner updates, and more

Fri, 22 May 2020 00:00:00 +0000

The last two weeks have included improvements to our ingredient search engine suggestions, meal planning improvements, and proposal of a product roadmap. We’d love feedback on these, and the roadmap in particular!

We made incremental progress on our internationalization processes (by trialing use of an application called poedit), but the process still involves some friction.

Support for viewing historic meal calendar entries remains on the near-term task list.

Partial Ingredient Name Suggestions

When you start typing ’tomato’ on RecipeRadar, the search engine will scan for suggestions across recipes that contain matching ingredient text, such as 1 tomato or two tomatoes (i.e. both plural and singular forms).

In this example, RecipeRadar suggests the plural name ’tomatoes’ to you, because the plural text appears more often in the recipe set than the singular form ’tomato'.

Until recently, these suggestions would only include ingredients where your input is a strict prefix of the ingredient name. So you might be offered “tomatoes” and “tomato sauce” as selections, but you would not be prompted with “cherry tomatoes” as an ingredient to search for.

This limitation (as reported a little over a month ago) has now been removed, making it easier to search for all kinds of ingredients and ingredient variants.

Have you encountered any problems finding ingredients or recipes on RecipeRadar? Let us know via the ‘Feedback’ button or by filing a GitHub issue and we’ll take a look.

Meal Planner Messaging

Some small changes have been applied to meal planning in the application this week.

While users can still create a shopping list without planning any meals, the ‘suggested action’ for each recipe search result now leads them towards meal planning rather than shopping list creation:

This aligns with our idea that users can save time, eat more healthily, and reduce food wastage by planning ahead for multiple meals.

To further attract users towards meal planning and to inform them about the state of their meal calendar, we have added a visual cue with the number of entries in the meal planner:

These small features will become increasingly valuable after we re-enable multi-device and multi-user session sharing.

Allowing family members and friends to view the state of meal planning and shopping at-a-glance, and providing access to the shopping list you created at home on your desktop computer while out with your smartphone are two of the cases we are considering.

Product Roadmap

Progress on RecipeRadar over the next few months should be methodical and sustainable; we have a large number of product goals, and while we can’t achieve them all instantly, we can arrange them thoughtfully so that we gradually deliver application improvements for users while retaining high quality and ensuring that our code remains easy to modify and extend, allowing us to reach our longer-term goals eventually.

To more easily communicate what those product goals are, and in what order we currently plan to implement them, we have drafted a RecipeRadar Product Roadmap. Please take a look and provide your feedback here or on the GitHub pull request.

Coming up next

As you’ll see on the roadmap, there are a number of architectural investments that we believe make sense to apply during the remainder of Q2 2020 (Apr to Jun 2020).

It’s likely that these will compose the bulk of our engineering work over the next two weeks; we may also take the opportunity to look into improving company transparency via OpenCollective and perhaps by increasing the external visibility of internal mailing lists.

As mentioned previously, adding the ability to view historic meal calendar entries in the application also remains on the near-term todo-list.

Open source ethics, ingredient highlighting, and entry to the Progressive Web App Store

Fri, 08 May 2020 00:00:00 +0000

Progress on RecipeRadar over the previous fortnight has included improvements to our data processing infrastructure, continued refinement of our screen prototypes, and exploration of collaboration tools and technology selections.

As you may have seen on the discussion list, Monica has been iterating on the screen designs for web and mobile, and in particular the prototype meal planner calendar is looking great. I’d like us all to spend a bit more time to make sure these designs are simple and usable. We should initially remove non-essential features, since we continue to have limited software development bandwidth at the moment.

Speaking of which: if you know any developers (senior or junior) who would be interested in learning about the project’s code and potentially getting involved further, please send them our way.

Does anyone have experience and/or opinions regarding the Flutter application framework? There’s a chance RecipeRadar may migrate to use it in future.

Organizational Goals

Today I was reading two articles that I’d like to share since it may help explain the direction I’d like to take the organization in, eventually, assuming that the application gains traction.

One article is ‘Open Sources: Voices from the Open Source Revolution’, and it is a record of the challenges and obstacles encountered by Cygnus, an early example of a business founded on open source software. When reading, please bear in mind that software is only one type of digital content – I believe the principles may apply to other digital creations.

While initially potentially seeming to conflict with the previous article – but in my opinion, equally based in the same reality – the second article is ‘The Ethics of Unpaid Labor and the OSS Community’ and it neatly describes an ongoing concern I have about driving the RecipeRadar project and asking for contributions while not necessarily providing much in return, yet.

Open digital environments should lend themselves well to contributions from diverse backgrounds, and should be something that people can contribute to from home and to build up a portfolio of work – or for pure enjoyment and interest in the field. And eventually the application should also provide value not just to ourselves as users, but also potentially in terms of funds to sustain continued contributions. In practice, opportunity is often equally distributed, and I’d welcome ideas to create a more inclusive and sustainable environment.

Ingredient and Direction Highlighting

One major benefit that RecipeRadar will provide compared to many existing recipe software tools is that it will understand information and context about the ingredients and nutrition in recipes, and will eventually – and always optionally – offer to help organize kitchen appliances and preparation timing. A simple recurring example I use is that it could identify that a recipe involves pre-heating an oven; and RecipeRadar could offer a reminder to let someone in your home know that it’s time to set the oven temperature.

As one of many steps towards this goal of a ‘smart and respectful’ kitchen assistant, ingredient and equipment metadata is now extracted from recipes, and formatting and punctuation of the original author’s intent are retained (thanks again, Citra).

To re-state the goal again here: we aim to add valuable additional information to the original recipe, and will always attribute and link back to the original source.

Here are some examples from a search for slow cooker lentil recipes earlier today - notice that the application has identified the matching ingredient, and highlighted the equipment used in the recipe directions.

PWA Store

Following submission two weeks ago, RecipeRadar has been accepted into the Progressive Web App Store!

This is great news and some people have already left reviews and our first review, featuring PB&J Sushi.

Coming up next

I mentioned in the last update that we’ll be looking for translations of messages that appear in the application soon. This is something we didn’t quite get around to over the previous two weeks, but it should be progressing within the next fortnight.

Now that the bulk of the backend data pipeline work for ingredient and direction highlighting is completed, focus can return to the frontend user interface and search improvements. Some items we’ll look at in the next two weeks include: partial name search, historic meal planner entries, and messaging improvements.

Updated search control designs, and de-duplication of recipes

Fri, 24 Apr 2020 00:00:00 +0000

The last two weeks have involved quality improvements, design prototyping, and preparatory work as we prepare to introduce more contributors to RecipeRadar.

Have you experienced any ingredient shortages or price changes during the coronavirus pandemic?

In future we hope that RecipeRadar will automatically offer information on local ingredient availability, seasonality, and pricing. This and other plans will be detailed in an upcoming roadmap document.

Design Iteration

As you may have seen on the mailing list, Monica has been preparing screens to provide an improved search experience in the application.

These are exciting changes and they provide the opportunity to make each RecipeRadar session more intuitive and rewarding.

Improvements

A long-standing issue that allowed duplicate recipes to appear in search results has been thoroughly solved!

The solution took some careful planning and implementation, and is documented for future reference.

If you see anything that looks unusual while using RecipeRadar, please feel free to report an issue via the ‘Feedback’ button in the app, or directly on GitHub.

Call for Developers

We’re looking for more software developers to help implement features and improvements for RecipeRadar.

Alongside the Code of Conduct mentioned in the last update, we now also have a set of Contribution Guidelines to help developers (and other contributors) get started in the most effective way possible.

If you know any software developers who are looking for projects to contribute to, please send them our way. Soon we’ll also be ready to invite help from anyone who can offer translations of the application into different languages.

Although we don’t currently offer financial compensation, we can offer attribution and credit for work provided, and I’d personally be glad to supply references for any work contributed to RecipeRadar when people are applying for jobs in future.

PWA Store

RecipeRadar has been submitted for inclusion in the Progressive Web App Store!

The PWA Store is similar to the iOS App Store and Android Play Store that you are more likely to be familiar with, and offers web-based applications that you can install for free on your own devices.

Coming up next

One of the features we’re most excited about is the inclusion of more ingredient context in recipe search results.

Where you would previously have seen a recipe call simply for onions, in future you’ll see the full ingredient description, such as large onions, diced.

The ingredient name will continue to be highlighted so that you can see at-a-glance how many ingredients match in each search result.

Thanks again to Citra for reporting this issue – it should soon be resolved thanks to a lot of preparatory work behind-the-scenes.

RecipeRadar

Fri, 10 Apr 2020 17:54:52 +0100

RecipeRadar aims to provide recipe search, meal planning, and cooking guidance to a global audience via services which are simple, intuitive, collaborative and effective, and always designed with users as our first priority.

We believe that eating well and making informed consumption decisions are a benefit to everyone, both individually and collectively.

We believe that the best technology is designed with users as the sole and primary benefactors, and that they should have the ability to inspect, modify and adjust that technology.

To meet these principles, we have made a number of intentional decisions about the structure of the company and our service:

RecipeRadar is developed by OpenCulinary C.I.C., a UK Community Interest Company. The use of this company structure locks in social goals and reduces the potential for growth or revenue pressures to lead the service in directions which negatively affect users. If you’re curious to learn more, please read our company documentation.
All of the code for our application and service is publicly available under the AGPLv3 license, and we gladly accept changes and contributions from our user community.
We do not use any proprietary software anywhere in the delivery of our service, which means that you can inspect and reproduce any part of our technology stack.
No personal data about you is stored on our servers; the only data transmitted from the application to the server is the data required to fulfill service requests. For example, when searching for recipes, the application sends the list of ingredients you specify so that the server can respond with suitable results.
Further to this, no cookies are stored in your browser, and no cookie data is sent to our servers.
When any data is transferred between the application and our servers, it is always encrypted with TLS. We use a strong cipher suite, and we test our configuration using SSL Labs’ ‘SSL Server Test’ tool to discover areas for improvement. If you discover or suspect any issue with our application or server security, please let us know and we’ll investigate.

Our first fortnightly update on the blog!

Fri, 10 Apr 2020 00:00:00 +0000

Progress on RecipeRadar has been ramping up over the past few days; here are some updates on recent developments.

Homepage Updates

Thanks to Susan and Liz, the homepage search controls have had a revamp:

All three of the recipe search fields are now shown by default, and the wording has been simplified and improved to cater to people finding ingredients unavailable due to the COVID-19 crisis.

User Experience Improvements

While more work remains to improve ingredient search, incremental progress is being made.

Handling of ’enter’ keypresses has been adjusted, with more improvements to follow.
Citra has reported an issue regarding ‘partial’ ingredient search and there is a plan to handle this in the RecipeRadar Search API.
A bugfix has been applied to ensure that pressing the search button always scrolls the user to see their search results. In future we plan to add automated testing to ensure that the bug can’t reappear.
A performance issue relating to recipe result lists was identified and fixed, leading to a more responsive search experience

Blog is Ready to Roll

The RecipeRadar blog is up and running, and posts can be added via the blog’s GitHub repository. If you’d feel like writing some content for the service - or know someone who would - please contact James with details.

Code of Conduct

An organization-wide Code of Conduct has been introduced, and aims to foster a comfortable and pleasant environment for discussion and collaboration.

If you have any concerns about anyone’s conduct relating to RecipeRadar, please contact us at conduct@reciperadar.com and we’ll address it.

Coming up next

There’s plenty more in store over the next couple of weeks.

We’ll be investigating improvements to link sharing so that people can more easily share recipe ideas they find with friends
There’s a user experience bug relating to ‘jumpy’ search results that we’re planning to get rid of
We’ll be upgrading the recipe crawling library that we use, and that will provide us with more sources of recipes from the web

Welcome!

Tue, 29 Oct 2019 13:13:48 -0700

The engineering tag will be the home for a series of articles detailing the engineering effort behind building RecipeRadar, a recipe search engine and meal planner.

From Kubernetes to proxy caching, from Python to ingredient graphs, we’ll write content to explain the technology behind the application, and how those technologies were selected and evaluated.