2019-10-04_Mastodon-3.0-API-and-deployment-changes.md 10.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156
title: "Mastodon 3.0 in-depth"
subtitle: "New APIs, deployment options, and admin settings"
draft: true
author: gargron
date: 2019-10-04
- mastodon
- api
## New REST APIs
### Profile directory

The profile directory is a way to discover users who want to be discovered. To fetch the profile directory, access `GET /api/v1/directory` with the possible params `local` (boolean) and `order` (`new` or `active`). Pagination is accomplished using `offset` and `limit` params.

### Trends

Hashtags that are used more than usual (and above a small minimal threshold) are "trending". To fetch trending hashtags, access `GET /api/v1/trends`. Only 10 results are returned maximally but you can request fewer with `limit` param.

### Managing featured hashtags

Users can feature hashtags on their public profile, which allows visitors to easily browse their public posts filed under those hashtags. These cannot yet be arbitrarily retrieved through the API, but there is now an API for managing the featured hashtags of the current user:

- `GET /api/v1/featured_tags` to retrieve current user's featured hashtags
- `POST /api/v1/featured_tags` to create a new featured hashtag, specified by the param `name`
- `DELETE /api/v1/featured_tags/:id` to delete a featured hashtag
- `GET /api/v1/featured_tags/suggestions` to retrieve the user's 10 most commonly used hashtags

A featured hashtag contains the attributes `id`, `name`, `statuses_count` and `last_status_at`.

### Timeline position markers

Apps can now synchronize their position in certain timelines between each other. Currently these are the home timeline and the notifications timeline. The web UI already implements this API and will save its position when closed.

To retrieve a map of markers with timeline names as keys, access `GET /api/v1/markers` . You must specify the desired timelines with the array param `timeline`. This is a slightly unusual structure in Mastodon's REST API so it deserves an example:

	"home": {
		"last_read_id": "123...",
		"updated_at": "2019-10-04...",
		"version": 1

	"notifications": {

To create a new marker, pass a map to `POST /api/v1/markers` with timeline names as keys (`home` and/or `notifications`), and an object containing the `last_read_id` for each timeline. Essentially, you pass it something like this, either encoded as JSON or using nested form/query params:

	"home": {
		"last_read_id": "567..."

### Hashtag autocomplete

If you are using the `GET /api/v2/search` API for showing the user autocomplete for hashtags, you can now pass the `exclude_unreviewed` boolean param to limit the results to only those hashtags that have been looked at by the server's staff. This is a way to reduce junk and harmful results.

### Sign-up API in approval-required registrations mode

You can now pass the `reason` param to `POST /api/v1/accounts`, containing the user's reason for wanting to join the server, which is useful when the server is in approval-required registrations mode. You can detect when that mode is active by the `approval_required` boolean attribute returned from `GET /api/v1/instance` (in conjunction with the `registrations` boolean attribute).

### Custom emoji categories

New attribute `category` on custom emojis returned from `GET /api/v1/custom_emojis` contains a string with which emojis are supposed to be grouped when displayed in a picker UI.

### Displaying user's own votes in polls

New attribute `own_votes` on polls contains an array of the user's choices (as indices corresponding to the `options` array).

## New search syntax support

When ElasticSearch is enabled, you can use the following syntax to fine-tune your search:

- Surround keywords with double quotes (`"`) to search for the exact phrase
- Prepend a keyword (or phrase) with minus sign (`-`) to exclude it from results

It should be noted that the default operator has been changed from "and" to "or", so by searching for "foo bar" you will get results that contain both "foo" and "bar" at the top, but also those that only contain "foo" and only contain "bar". For this reason, there is also another new operator, the plus sign (`+`) which you can prepend to a keyword or phrase to make sure the results definitely contain it.

## Health check

There is now `GET /health` endpoint for the web process which you can use with a monitoring service. The endpoint measures not only that the web process responds to requests but can successfully connect to the database and the cache as well.

## New deployment settings
### Reply-to header on e-mails

If you want e-mails to be sent with a reply-to header, i.e. redirecting replies to those e-mails to a particular address, use the new `SMTP_REPLY_TO` environment variable. Mind that the reply-to header on moderation warning e-mails is set to the contact address configured in the admin UI.

### Secure mode

Normally, all public resources are available without authentication or authorization. Because of this, it is hard to know who (in particular, which server, or which person) has accessed a particular resource, and impossible to deny that access to the ones you want to avoid. Secure mode requires authentication (via HTTP signatures) on all public resources, as well as disabling public REST API access (i.e. no access without access token, and no access with app-only access tokens, there has to be a user assigned to that access token). This means you always know who is accessing *any* resource on your server, and can deny that access using domain blocks.

Unfortunately, secure mode is not fully backwards-compatible with previous Mastodon versions. For this reason, it cannot be enabled by default. If you want to enable it, knowing that it may negatively impact communications with other servers, set the `AUTHORIZED_FETCH=true` environment variable.

### Whitelist mode

Taking a step further than the secure mode, whitelist mode is meant for private servers. Our aim here are educational uses, such as schools and universities, where Mastodon could be used to provide a safe learning environment. When whitelist mode is enabled, no page is available without login, and any incoming or outgoing federation is ignored except for manually whitelisted domains. Domains can be whitelisted in the federation part of the admin UI. When whitelist mode is enabled, secure mode is also enabled.

To enable whitelist mode, set the `WHITELIST_MODE=true` environment variable. Please mind that this option was not designed for being switched on on already running servers. To clean an existing database of content that is not whitelisted, run `tootctl domains purge --whitelist-mode`

Because whitelist mode essentially creates a silo, not unlike Twitter, Facebook, and other centralized services, we do not recommend running public servers in whitelist mode.

## New command-line tools

Please mind that if you find any of the below descriptions insufficient, you can always append `--help` to whichever command you're interested in and receive the most detailed information about the usage of that command and the available options.

### Parallization and progress

Commands that used to accept a `--background` flag for Sidekiq-based execution have been changed to instead support a `--concurrency` (or `-c`) flag specifying the number of threads to use for parallel execution.

Instead of printing dots to signal progress, real progress bars are now displayed, with the number of processed items and estimated time to completion.

### Cleaning up old link preview cards

To remove thumbnails from older link preview cards, run `tootctl preview_cards remove`, specifying age with `--days` just like for media removal.

### Re-downloading removed media attachments

If you need to re-download media attachments, run `tootctl media refresh`. You can either re-download media attachments from a specific `--status`, from a specific `--account`, or from an entire `--domain`.

### Re-counting counters

Sometimes various counters in Mastodon get out of sync with reality. To fix account counters (e.g. followers, following, toots), run `tootctl cache recount accounts`. This should not take very long. To fix status counters (e.g. reblogs, favourites, replies), run `tootctl cache recount statuses`. This may take a lot longer.

## New admin UIs
### Trends

Hashtags will not trend without your approval. Whenever a hashtag is beginning to trend, you receive a notification e-mail asking to review it. You can disable those e-mails from your personal e-mail notification preferences. You can disable the trends feature altogether from admin settings.

The hashtags area in the admin UI has been updated. When looking at hashtags that are pending review, you can approve or reject them in batches. From individual hashtag view, you can control whether the hashtag can trend, whether it can appear on the profile directory and in searches, or whether it can be used at all. You will also see which servers you know about are contributing how much to that hashtag's usage to help you determine whether to let it trend or not.

### Including reported toots in warning e-mails

If you want to perform an action or warning against a user related to a report, you can choose if the toots that were in that report should be included in the e-mail the user will get about that action or warning. This will provide more clarity to the user about how they broke your rules.

### Table of contents on about page

The about page of your server will now auto-generate a table of contents based on the structure of your extended description HTML. It is recommended to have a `h1` tag, which will not be reflected on the table of contents, to give the entire page a title, then `h2` and `h3` tags for the different sections. Make sure your HTML is valid, otherwise the table of contents may not work as expected.

### Public and private domain blocks information

You can now add comments to domain blocks. Private comments are for other staff members only. From the admin settings, you can choose if domain blocks should be disclosed publicly or to logged-in users only, or not at all. If you choose to disclose them, they will appear on the about page, below your extended description. You can use the public comments to give public reasons for your decisions.

### Custom emoji categories

The custom emojis area in the admin UI has been updated. You can now assign emojis to custom categories and perform batch actions on them such as copying, deleting, or unlisting.

### Spam checks

When a user mentions someone who isn't following them *and* it's not a reply to something directed at that user, their message is run through a simplistic spam check which detects repeating messages. When spam is detected, the offending user is auto-silenced and a new report is created automatically. If that was a mistake, you can manually unsilence the user, and they will be exempt from future spam checks. You can disable the spam check feature from admin settings.