Home on Leandro Favarin

Scheduling Scripts on macOS with launchd

Mon, 03 Feb 2025 00:00:00 +0000

While I’ve used Google Photos extensively as a central storage for my personal photos, Apple Photos has recently taken over this responsibility. This article explains how I replaced an always-on service that archived my Google Photos with a minimally viable macOS launchd agent for Apple Photos.

It’s great that we can rely on Google and Apple to keep our photos as a one copy, one storage medium, and one off-site location, but that’s not enough to meet the “3–2–1 rule”. We also need local backups that we’re continuously synchronizing to act as a second copy and a second storage medium.

Apple Photos exists within the confines of iCloud and, as a consequence, if you want to automate backups in the same way that you can with GPhotos, you need to provide projects like this or this your iCloud password. With Jake Wharton’s docker-gphotos-sync project, you are free to use your real password, or (preferably) create an app-specific password, sandboxing potential incidents from spreading further. There does not seem to exist a similar tool for Apple Photos. Although I would prefer a more versatile and reliable tool, hacking something fast was more important.

Since I have the entirety of my library local, I could define a job on my crontab to clone its contents to another medium using standard CLI tools. However, cron is no longer the recommended approach for running headless tasks on a Mac. Per Apple’s documentation:

If you are running per-user background processes for OS X, launchd is also the preferred way to start these processes. These per-user processes are referred to as user agents. A user agent is essentially identical to a daemon, but is specific to a given logged-in user and executes only while that user is logged in.

The scheduling method used in macOS is not only preferable, but also significantly better than traditional approaches such as cron jobs. For example, if a computer is asleep when a task is scheduled to run, macOS will automatically reschedule the job to execute once the system wakes up. In contrast, cron jobs are completely ignored and only run when the computer is awake.

It’s not particularly complicated to set up a hello world for a user’s launch agent (example), so I started with a simple shell script that invoked rsync to mirror my Photos Library directory to my NAS. But recent security improvements in macOS have tightened the system up to a point where much of the internet literature became outdated. An example is not being able to access user’s directories (such as ~/Pictures or ~/Documents) via terminal emulators unless they’ve been given the Full Disk Access permission. The specific error I got was rsync: opendir "~/Pictures/Photo Library" failed: Permission denied (13).

A workaround would be to disable SIP or allow bash/zsh FDA, but a much safer approach would be to allow only rsync access these directories. Unfortunately, even this did not solve the issue, as confirmed by several developers in the Stack Overflow thread linked above. A suggested workaround was to create a dedicated application that runs the script and give it FDA. This approach worked for me on macOS Sequoia 15.3.

In short, you want the following:

An application (I used Automator.app to create one) that runs a shell script that executes rsync with the correct arguments.
A .plist file inside ~/Library/LaunchAgents/ that invokes this application ¹.

In my case, the application contained ²:

#!/bin/sh
/usr/bin/rsync -azv --delete-after \
 "/Users/leandro/Pictures/Photos Library.photoslibrary/" \
 mynas:\''Photos Library.photoslibrary/'\'

With local.PhotoSync.plist at ~/Library/LaunchAgents/ being:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>local.PhotoSync</string>
 <key>ProgramArguments</key>
 <array>
 <string>/usr/bin/open</string>
 <string>/Users/leandro/Applications/PhotoSync.app</string>
 </array>
 <key>StartCalendarInterval</key>
 <dict>
 <key>Hour</key>
 <integer>0</integer>
 <key>Minute</key>
 <integer>0</integer>
 </dict>
</dict>
</plist>

to run at every midnight, as specified with StartCalendarInterval. With them in place, the commands to register³ & unregister the agent on launchd are:

launchctl load -k <your-plist>

launchctl unload <your-plist>

The man pages of launchctl and launchd.plist are fairly easy to understand and cover a lot of details not mentioned in this article.

By setting an automated, in-background process that constantly replicates my photos to a secure offsite storage, I’ve removed Apple as a single point of failure in the event of unexpected account closure.

I believe that the security and preservation of our digital memories should be a top priority and were we must actively more actions than to rely on a free service to magically exist and serve us.

The command plutil -lint <plist-file> can be used to validate the syntax. ↩︎
The pair of single quotes above is necessary to preserve the space of the destination path, which is interpreted on the target machine. ↩︎
the -k argument kills any running instance before restarting the service. ↩︎

Drive your UI with SQLDelight’s views

Sun, 22 Sep 2019 00:00:00 +0000

Users demand more from apps every year and 2019 feels as though this pressure has never been so high. Among many ways we can improve the user experience as developers, driving the interface from a local database is an easy way to achieve robustness and flexibility. This is especially the case with database tools that can broadcast changes in the data set to interested parties.

Suppose you’re writing a database-driven application using SQLDelight, and you need to present the same type of data in slightly different ways. It could be that they differ on how they’re ordered, or just a subset of them.

SQLDelight automatically generates model objects for every query that you write. In many cases, the types it generates are either from a single column (SELECT id FROM Foo) or from the star projection (SELECT * FROM Foo). The former being simple Kotlin String/Long/Int types, and the latter the type SQLDelight generates from your table definition. What happens, then, if you select a subset of the properties declared on your table?

In this case SQLDelight will create a new type for every query, with its name being the named query:

bandsOrderedByName:
SELECT id, name
FROM band
ORDER BY name DESC;

bandsOrderedByAge:
SELECT id, name
FROM band
ORDER BY age;

A simplified code of what SQLDelight generates is:

data class BandsOrderedByName(id: String, name: String)

data class BandsOrderedByAge(id: String, name: String)

If you wanted to transform them later on, you would have to write two functions of types (BandsOrderedByName) -> X and (BandsOrderedByAge) -> X, even though the underlying composition is the same: two Strings.

While this is a simple exercise, in real-life examples projections tend to be more complex, usually with join clauses and many properties selected. For example:

SELECT
 band.id, 
 band.name,
 album.*
FROM band
JOIN album ON band.id = album.band_id;

A SQL View solves this problem elegantly, with our queries now being:

CREATE VIEW bandWithAlbum AS
SELECT
 band.id, 
 band.name,
 album.*
FROM band
JOIN album ON band.id = album.band_id; 

bandsOrderedByName:
SELECT *
FROM bandWithAlbum
ORDER BY name DESC;

bandsOrderedByAge:
SELECT *
FROM bandWithAlbum
ORDER BY age;

SQLDelight will generate BandWithAlbum type that can be queried, ordered, and filtered differently, per query. Much of the SELECT boilerplate from the queries is eliminated while still allowing for specific constraints, additional joins for scoping, as well as selecting as a list or as a single item. Moreover, it creates a solid building block for pagination, with:

count:
SELECT count(*)
FROM bandWithAlbum;

paged:
SELECT *
FROM bandWithAlbum
LIMIT ?
OFFSET ?;

Minimal effort is then needed to present items to the user by using only one RecyclerView.Adapter.

A combination of any out-of-the-box extension such as for Kotlin Coroutines or RxJava plus DiffUtil allows the user to easily visualize the data the way they want, because since the model being generated is a value class, the diff callbacks can be trivially written. One example would be:

object BandItemCallback : ItemCallback<BandWithAlbum>() {
 override fun areItemsTheSame(oldItem: BandWithAlbum, newItem: BandWithAlbum): Boolean {
 return oldItem.id == newItem.id
 }

 override fun areContentsTheSame(oldItem: BandWithAlbum, newItem: BandWithAlbum): Boolean {
 return oldItem == newItem
 }
}

Going back to the queries with different sorting, one could use an enum class to model the sort options and use Kotlin’s when expression to write:

enum class Sort { NAME, AGE }

fun bandsSorted(by: Sort): Flow<List<BandWithAlbum>> = when (by) {
 NAME -> db.bandsOrderedByName()
 AGE -> db.bandsOrderedByAge()
}.asFlow().mapToList()

and delegate to SQL algorithms that would be less efficient to run if they were executed programmatically.

Building views for the types you want and then querying those views for the data you need is an easy way to attain responsiveness and flexibility with very little code needed. It’s clear that user demands will only get more intense as technology develops. It’s therefore crucial to stay on top of these simple ways to improve the experience without breaking our backs (or keyboards).

Many thanks to Jake Wharton and Alec Strong for proofreading this article!

Don't fear SQL - A better way to store and handle data with SQLBrite & SQLDelight

Fri, 14 Jul 2017 00:00:00 +0000

For many people, SQL can be intimidating when writing mobile apps. To avoid its complexity, various libraries started to implement techniques such as Object-Relational Mapping. They ease basic operations on complex objects, but come with a set of downsides such as decreased performance and the learning curve of a new library.

Square’s libraries SQLBrite and SQLDelight improve data manipulation by embracing all the powerful capabilities of SQLite while removing common frictions like runtime crashes, boilerplate code, and type-unsafe APIs.

In this talk I’ll present the reactive mindset behind SQLBrite and the code-generation capabilities of SQLDelight. When combined, these two libraries will help you architect and code safer and faster with queries autocompletion, code reuse, and much more.

Exponential-Backoff RxJava operator with Jitter

Sat, 11 Mar 2017 00:00:00 +0000

Stripe recently published a technical article about how they handle errors between the clients and the server. The whole post is worth a read, and one of the topics mentioned was “Being a good distributed citizen”, in which mobile clients play fair when receiving failures from network operations. From their article (emphasis mine):

It’s usually recommended that clients follow something akin to an exponential backoff algorithm as they see errors. The client blocks for a brief initial wait time on the first failure, but as the operation continues to fail, it waits proportionally to 2^n, where n is the number of failures that have occurred. By backing off exponentially, we can ensure that clients aren’t hammering on a downed server and contributing to the problem.

Exponential backoff has a long and interesting history in computer networking.

Furthermore, it’s also a good idea to mix in an element of randomness. If a problem with a server causes a large number of clients to fail at close to the same time, then even with back off, their retry schedules could be aligned closely enough that the retries will hammer the troubled server. This is known as the thundering herd problem.

We can address thundering herd by adding some amount of random “jitter” to each client’s wait time. This will space out requests across all clients, and give the server some breathing room to recover.

Network failures are everywhere and can happen at any time, and if you have several clients failing at the same time, it is a BIG deal.

Since I couldn’t find any flexible yet powerful solution for this problem written in Kotlin I decided to write my own using RxJava 2. The operator is:

/**
 * Exponential backoff that respects the equation: delay * retries ^ 2 * jitter
 */
class ExpBackoff(
 private val jitter: Jitter,
 private val delay: Long,
 private val unit: TimeUnit,
 private val retries: Int = 0
) : Function<Observable<out Throwable>, Observable<Long>> {

 @Throws(Exception::class)
 override fun apply(observable: Observable<out Throwable>): Observable<Long> {
 return observable
 .zipWith(Observable.range(1, retries), BiFunction<Throwable, Int, Int> { _, retryCount ->
 retryCount
 })
 .flatMap { attemptNumber -> Observable.timer(getNewInterval(attemptNumber), unit) }
 }

 private fun getNewInterval(retryCount: Int): Long {
 var newInterval = (delay * Math.pow(retryCount.toDouble(), 2.0) * jitter.get()).toLong()
 if (newInterval < 0) {
 newInterval = Long.MAX_VALUE
 }
 return newInterval
 }
}

With Jitter being:

interface Jitter {
 fun get(): Double

 companion object {
 val NO_OP = { 1 }
 }
}

A default implementation that could deviate up to 15% could be written as:

class DefaultJitter : Jitter {
 private val random = Random()

 /** Returns a random value inside [0.85, 1.15] every time it's called */
 override fun get(): Double = 0.85 + random.nextDouble() % 0.3f
}

The implementation here is not 1:1 to what Stripe did, but it could be changed easily to adapt to your needs.

Its usage is then very simple. Just apply it before subscribing:

observable
 ...
 .retryWhen(ExpBackoff(DefaultJitter(), delay = 1, unit = SECONDS, retries = 3))
 .subscribe(/* */)