Merge pull request #547 from kmycode/kb-draft-10.4

Release: 10.4

.bundler-audit.yml

@@ -0,0 +1,6 @@
+---
+ignore:
+  # devise-two-factor advisory about brute-forcing TOTP
+  # We have rate-limits on authentication endpoints in place (including second
+  # factor verification) since Mastodon v3.2.0
+  - CVE-2024-0227

.ruby-version

@@ -1 +1 @@
-3.2.2
+3.2.3

CHANGELOG.md

@@ -2,6 +2,65 @@
 
 All notable changes to this project will be documented in this file.
 
+## [4.2.4] - 2024-01-24
+
+### Fixed
+
+- Fix error when processing remote files with unusually long names ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28823))
+- Fix processing of compacted single-item JSON-LD collections ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28816))
+- Retry 401 errors on replies fetching ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/28788))
+- Fix `RecordNotUnique` errors in LinkCrawlWorker ([tribela](https://github.com/mastodon/mastodon/pull/28748))
+- Fix Mastodon not correctly processing HTTP Signatures with query strings ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28443), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28476))
+- Fix potential redirection loop of streaming endpoint ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28665))
+- Fix streaming API redirection ignoring the port of `streaming_api_base_url` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28558))
+- Fix error when processing link preview with an array as `inLanguage` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28252))
+- Fix unsupported time zone or locale preventing sign-up ([Gargron](https://github.com/mastodon/mastodon/pull/28035))
+- Fix "Hide these posts from home" list setting not refreshing when switching lists ([brianholley](https://github.com/mastodon/mastodon/pull/27763))
+- Fix missing background behind dismissable banner in web UI ([Gargron](https://github.com/mastodon/mastodon/pull/27479))
+- Fix line wrapping of language selection button with long locale codes ([gunchleoc](https://github.com/mastodon/mastodon/pull/27100), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27127))
+- Fix `Undo Announce` activity not being sent to non-follower authors ([MitarashiDango](https://github.com/mastodon/mastodon/pull/18482))
+- Fix N+1s because of association preloaders not actually getting called ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28339))
+- Fix empty column explainer getting cropped under certain conditions ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28337))
+- Fix `LinkCrawlWorker` error when encountering empty OEmbed response ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28268))
+- Fix call to inefficient `delete_matched` cache method in domain blocks ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28367))
+
+### Security
+
+- Add rate-limit of TOTP authentication attempts at controller level ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28801))
+
+## [4.2.3] - 2023-12-05
+
+### Fixed
+
+- Fix dependency on `json-canonicalization` version that has been made unavailable since last release
+
+## [4.2.2] - 2023-12-04
+
+### Changed
+
+- Change dismissed banners to be stored server-side ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27055))
+- Change GIF max matrix size error to explicitly mention GIF files ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27927))
+- Change `Follow` activities delivery to bypass availability check ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/27586))
+- Change single-column navigation notice to be displayed outside of the logo container ([renchap](https://github.com/mastodon/mastodon/pull/27462), [renchap](https://github.com/mastodon/mastodon/pull/27476))
+- Change Content-Security-Policy to be tighter on media paths ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/26889))
+- Change post language code to include country code when relevant ([gunchleoc](https://github.com/mastodon/mastodon/pull/27099), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27207))
+
+### Fixed
+
+- Fix upper border radius of onboarding columns ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27890))
+- Fix incoming status creation date not being restricted to standard ISO8601 ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27655), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28081))
+- Fix some posts from threads received out-of-order sometimes not being inserted into timelines ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27653))
+- Fix posts from force-sensitized accounts being able to trend ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27620))
+- Fix error when trying to delete already-deleted file with OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27569))
+- Fix batch attachment deletion when using OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27554))
+- Fix processing LDSigned activities from actors with unknown public keys ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27474))
+- Fix error and incorrect URLs in `/api/v1/accounts/:id/featured_tags` for remote accounts ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27459))
+- Fix report processing notice not mentioning the report number when performing a custom action ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27442))
+- Fix handling of `inLanguage` attribute in preview card processing ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27423))
+- Fix own posts being removed from home timeline when unfollowing a used hashtag ([kmycode](https://github.com/mastodon/mastodon/pull/27391))
+- Fix some link anchors being recognized as hashtags ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27271), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27584))
+- Fix format-dependent redirects being cached regardless of requested format ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27634))
+
 ## [4.2.1] - 2023-10-10
 
 ### Added

Dockerfile

@@ -7,15 +7,15 @@
 ARG TARGETPLATFORM=${TARGETPLATFORM}
 ARG BUILDPLATFORM=${BUILDPLATFORM}
 
-# Ruby image to use for base image, change with [--build-arg RUBY_VERSION="3.2.2"]
-ARG RUBY_VERSION="3.2.2"
+# Ruby image to use for base image, change with [--build-arg RUBY_VERSION="3.2.3"]
+ARG RUBY_VERSION="3.2.3"
 # # Node version to use in base image, change with [--build-arg NODE_MAJOR_VERSION="20"]
 ARG NODE_MAJOR_VERSION="20"
 # Debian image to use for base image, change with [--build-arg DEBIAN_VERSION="bookworm"]
 ARG DEBIAN_VERSION="bookworm"
 # Node image to use for base image based on combined variables (ex: 20-bookworm-slim)
 FROM docker.io/node:${NODE_MAJOR_VERSION}-${DEBIAN_VERSION}-slim as node
-# Ruby image to use for base image based on combined variables (ex: 3.2.2-slim-bookworm)
+# Ruby image to use for base image based on combined variables (ex: 3.2.3-slim-bookworm)
 FROM docker.io/ruby:${RUBY_VERSION}-slim-${DEBIAN_VERSION} as ruby
 
 # Resulting version string is vX.X.X-MASTODON_VERSION_PRERELEASE+MASTODON_VERSION_METADATA

Gemfile.lock

@@ -481,8 +481,9 @@ GEM
       timeout
     net-smtp (0.4.0)
       net-protocol
-    nio4r (2.5.9)
-    nokogiri (1.16.0)
+    net-ssh (7.1.0)
+    nio4r (2.7.0)
+    nokogiri (1.16.2)
       mini_portile2 (~> 2.8.2)
       racc (~> 1.4)
     oj (3.16.3)
@@ -544,7 +545,7 @@ GEM
     psych (5.1.2)
       stringio
     public_suffix (5.0.4)
-    puma (6.4.1)
+    puma (6.4.2)
       nio4r (~> 2.0)
     pundit (2.3.1)
       activesupport (>= 3.0.0)
@@ -726,7 +727,7 @@ GEM
       rufus-scheduler (~> 3.2)
       sidekiq (>= 6, < 8)
       tilt (>= 1.4.0)
-    sidekiq-unique-jobs (7.1.30)
+    sidekiq-unique-jobs (7.1.33)
       brpoplpush-redis_script (> 0.1.1, <= 2.0.0)
       concurrent-ruby (~> 1.0, >= 1.0.5)
       redis (< 5.0)

app/controllers/activitypub/references_controller.rb

@@ -5,8 +5,6 @@ class ActivityPub::ReferencesController < ActivityPub::BaseController
   include Authorization
   include AccountOwnedConcern
 
-  REFERENCES_LIMIT = 5
-
   before_action :require_signature!, if: :authorized_fetch_mode?
   before_action :set_status
 
@@ -40,17 +38,21 @@ class ActivityPub::ReferencesController < ActivityPub::BaseController
     @results ||= begin
       references = @status.reference_objects.order(target_status_id: :asc)
       references = references.where('target_status_id > ?', page_params[:min_id]) if page_params[:min_id].present?
-      references = references.limit(limit_param(REFERENCES_LIMIT))
+      references = references.limit(limit_param(references_limit))
       references.pluck(:target_status_id)
     end
   end
 
+  def references_limit
+    StatusReference::REFERENCES_LIMIT
+  end
+
   def pagination_min_id
     results.last
   end
 
   def records_continue?
-    results.size == limit_param(REFERENCES_LIMIT)
+    results.size == limit_param(references_limit)
   end
 
   def references_collection_presenter

app/controllers/api/v1/streaming_controller.rb

@@ -2,7 +2,7 @@
 
 class Api::V1::StreamingController < Api::BaseController
   def index
-    if Rails.configuration.x.streaming_api_base_url == request.host
+    if same_host?
       not_found
     else
       redirect_to streaming_api_url, status: 301, allow_other_host: true
@@ -11,6 +11,11 @@ class Api::V1::StreamingController < Api::BaseController
 
   private
 
+  def same_host?
+    base_url = Addressable::URI.parse(Rails.configuration.x.streaming_api_base_url)
+    request.host == base_url.host && request.port == (base_url.port || 80)
+  end
+
   def streaming_api_url
     Addressable::URI.parse(request.url).tap do |uri|
       base_url = Addressable::URI.parse(Rails.configuration.x.streaming_api_base_url)

app/controllers/auth/omniauth_callbacks_controller.rb

@@ -7,7 +7,7 @@ class Auth::OmniauthCallbacksController < Devise::OmniauthCallbacksController
   def self.provides_callback_for(provider)
     define_method provider do
       @provider = provider
-      @user = User.find_for_oauth(request.env['omniauth.auth'], current_user)
+      @user = User.find_for_omniauth(request.env['omniauth.auth'], current_user)
 
       if @user.persisted?
         record_login_activity
@@ -17,6 +17,9 @@ class Auth::OmniauthCallbacksController < Devise::OmniauthCallbacksController
         session["devise.#{provider}_data"] = request.env['omniauth.auth']
         redirect_to new_user_registration_url
       end
+    rescue ActiveRecord::RecordInvalid
+      flash[:alert] = I18n.t('devise.failure.omniauth_user_creation_failure') if is_navigational_format?
+      redirect_to new_user_session_url
     end
   end
 

app/controllers/auth/sessions_controller.rb

@@ -1,6 +1,10 @@
 # frozen_string_literal: true
 
 class Auth::SessionsController < Devise::SessionsController
+  include Redisable
+
+  MAX_2FA_ATTEMPTS_PER_HOUR = 10
+
   layout 'auth'
 
   skip_before_action :check_self_destruct!
@@ -130,9 +134,23 @@ class Auth::SessionsController < Devise::SessionsController
     session.delete(:attempt_user_updated_at)
   end
 
+  def clear_2fa_attempt_from_user(user)
+    redis.del(second_factor_attempts_key(user))
+  end
+
+  def check_second_factor_rate_limits(user)
+    attempts, = redis.multi do |multi|
+      multi.incr(second_factor_attempts_key(user))
+      multi.expire(second_factor_attempts_key(user), 1.hour)
+    end
+
+    attempts >= MAX_2FA_ATTEMPTS_PER_HOUR
+  end
+
   def on_authentication_success(user, security_measure)
     @on_authentication_success_called = true
 
+    clear_2fa_attempt_from_user(user)
     clear_attempt_from_session
 
     user.update_sign_in!(new_sign_in: true)
@@ -164,4 +182,8 @@ class Auth::SessionsController < Devise::SessionsController
       user_agent: request.user_agent
     )
   end
+
+  def second_factor_attempts_key(user)
+    "2fa_auth_attempts:#{user.id}:#{Time.now.utc.hour}"
+  end
 end

app/controllers/concerns/auth/two_factor_authentication_concern.rb

@@ -66,6 +66,11 @@ module Auth::TwoFactorAuthenticationConcern
   end
 
   def authenticate_with_two_factor_via_otp(user)
+    if check_second_factor_rate_limits(user)
+      flash.now[:alert] = I18n.t('users.rate_limited')
+      return prompt_for_two_factor(user)
+    end
+
     if valid_otp_attempt?(user)
       on_authentication_success(user, :otp)
     else

app/controllers/concerns/signature_verification.rb

@@ -266,7 +266,7 @@ module SignatureVerification
       stoplight_wrap_request { ResolveAccountService.new.call(key_id.delete_prefix('acct:'), suppress_errors: false) }
     elsif !ActivityPub::TagManager.instance.local_uri?(key_id)
       account   = ActivityPub::TagManager.instance.uri_to_actor(key_id)
-      account ||= stoplight_wrap_request { ActivityPub::FetchRemoteKeyService.new.call(key_id, id: false, suppress_errors: false) }
+      account ||= stoplight_wrap_request { ActivityPub::FetchRemoteKeyService.new.call(key_id, suppress_errors: false) }
       account
     end
   rescue Mastodon::PrivateNetworkAddressError => e

app/helpers/jsonld_helper.rb

@@ -163,8 +163,8 @@ module JsonLdHelper
     end
   end
 
-  def fetch_resource(uri, id, on_behalf_of = nil)
-    unless id
+  def fetch_resource(uri, id_is_known, on_behalf_of = nil, request_options: {})
+    unless id_is_known
       json = fetch_resource_without_id_validation(uri, on_behalf_of)
 
       return if !json.is_a?(Hash) || unsupported_uri_scheme?(json['id'])
@@ -172,14 +172,14 @@ module JsonLdHelper
       uri = json['id']
     end
 
-    json = fetch_resource_without_id_validation(uri, on_behalf_of)
+    json = fetch_resource_without_id_validation(uri, on_behalf_of, request_options: request_options)
     json.present? && json['id'] == uri ? json : nil
   end
 
-  def fetch_resource_without_id_validation(uri, on_behalf_of = nil, raise_on_temporary_error = false)
+  def fetch_resource_without_id_validation(uri, on_behalf_of = nil, raise_on_temporary_error = false, request_options: {})
     on_behalf_of ||= Account.representative
 
-    build_request(uri, on_behalf_of).perform do |response|
+    build_request(uri, on_behalf_of, options: request_options).perform do |response|
       raise Mastodon::UnexpectedResponseError, response unless response_successful?(response) || response_error_unsalvageable?(response) || !raise_on_temporary_error
 
       body_to_json(response.body_with_limit) if response.code == 200
@@ -212,8 +212,8 @@ module JsonLdHelper
     response.code == 501 || ((400...500).cover?(response.code) && ![401, 408, 429].include?(response.code))
   end
 
-  def build_request(uri, on_behalf_of = nil)
-    Request.new(:get, uri).tap do |request|
+  def build_request(uri, on_behalf_of = nil, options: {})
+    Request.new(:get, uri, **options).tap do |request|
       request.on_behalf_of(on_behalf_of) if on_behalf_of
       request.add_headers('Accept' => 'application/activity+json, application/ld+json')
     end

app/javascript/mastodon/features/compose/components/privacy_dropdown.jsx

@@ -169,6 +169,7 @@ class PrivacyDropdown extends PureComponent {
     value: PropTypes.string.isRequired,
     onChange: PropTypes.func.isRequired,
     noDirect: PropTypes.bool,
+    noLimited: PropTypes.bool,
     replyToLimited: PropTypes.bool,
     container: PropTypes.func,
     disabled: PropTypes.bool,
@@ -260,6 +261,10 @@ class PrivacyDropdown extends PureComponent {
       );
     }
 
+    if (this.props.noLimited) {
+      this.options = this.options.filter((opt) => !['mutual', 'circle'].includes(opt.value));
+    }
+
     this.selectableOptions = [...this.options];
 
     if (!enableLoginPrivacy) {

app/javascript/mastodon/features/ui/components/boost_modal.jsx

@@ -110,6 +110,7 @@ class BoostModal extends ImmutablePureComponent {
           {status.get('visibility') !== 'private' && !status.get('reblogged') && (
             <PrivacyDropdown
               noDirect
+              noLimited
               value={privacy}
               container={this._findContainer}
               onChange={this.props.onChangeBoostPrivacy}

app/lib/account_statuses_filter.rb

@@ -33,7 +33,8 @@ class AccountStatusesFilter
     available_visibilities -= [:unlisted] if (domain_block&.detect_invalid_subscription || misskey_software?) && @account.user&.setting_reject_unlisted_subscription
     available_visibilities -= [:login] if current_account.nil?
 
-    scope.merge!(scope.where(spoiler_text: ['', nil])) if domain_block&.reject_send_sensitive
+    scope.merge!(scope.where(sensitive: false)) if domain_block&.reject_send_sensitive
+
     scope.merge!(scope.where(searchability: available_searchabilities))
     scope.merge!(scope.where(visibility: available_visibilities))
 
@@ -153,9 +154,9 @@ class AccountStatusesFilter
   end
 
   def domain_block
-    return nil if @account.nil? || @account.local?
+    return nil if @current_account.nil? || @current_account.local?
 
-    @domain_block = DomainBlock.find_by(domain: @account.domain)
+    @domain_block = DomainBlock.find_by(domain: @current_account.domain)
   end
 
   def misskey_software?

app/lib/activitypub/activity.rb

@@ -154,7 +154,7 @@ class ActivityPub::Activity
     if object_uri.start_with?('http')
       return if ActivityPub::TagManager.instance.local_uri?(object_uri)
 
-      ActivityPub::FetchRemoteStatusService.new.call(object_uri, id: true, on_behalf_of: @account.followers.local.first, request_id: @options[:request_id])
+      ActivityPub::FetchRemoteStatusService.new.call(object_uri, on_behalf_of: @account.followers.local.first, request_id: @options[:request_id])
     elsif @object['url'].present?
       ::FetchRemoteStatusService.new.call(@object['url'], request_id: @options[:request_id])
     end

app/lib/activitypub/linked_data_signature.rb

@@ -19,7 +19,7 @@ class ActivityPub::LinkedDataSignature
     return unless type == 'RsaSignature2017'
 
     creator = ActivityPub::TagManager.instance.uri_to_actor(creator_uri)
-    creator = ActivityPub::FetchRemoteKeyService.new.call(creator_uri, id: false) if creator&.public_key.blank?
+    creator = ActivityPub::FetchRemoteKeyService.new.call(creator_uri) if creator&.public_key.blank?
 
     return if creator.nil?
 

app/lib/application_extension.rb

@@ -4,14 +4,34 @@ module ApplicationExtension
   extend ActiveSupport::Concern
 
   included do
+    include Redisable
+
     has_many :created_users, class_name: 'User', foreign_key: 'created_by_application_id', inverse_of: :created_by_application
 
     validates :name, length: { maximum: 60 }
     validates :website, url: true, length: { maximum: 2_000 }, if: :website?
     validates :redirect_uri, length: { maximum: 2_000 }
+
+    # The relationship used between Applications and AccessTokens is using
+    # dependent: delete_all, which means the ActiveRecord callback in
+    # AccessTokenExtension is not run, so instead we manually announce to
+    # streaming that these tokens are being deleted.
+    before_destroy :push_to_streaming_api, prepend: true
   end
 
   def confirmation_redirect_uri
     redirect_uri.lines.first.strip
   end
+
+  def push_to_streaming_api
+    # TODO: #28793 Combine into a single topic
+    payload = Oj.dump(event: :kill)
+    access_tokens.in_batches do |tokens|
+      redis.pipelined do |pipeline|
+        tokens.ids.each do |id|
+          pipeline.publish("timeline:access_token:#{id}", payload)
+        end
+      end
+    end
+  end
 end

app/lib/status_reach_finder.rb

@@ -38,35 +38,26 @@ class StatusReachFinder
   private
 
   def reached_account_inboxes
+    Account.where(id: reached_account_ids).where.not(domain: banned_domains).inboxes
+  end
+
+  def reached_account_inboxes_for_misskey
+    Account.where(id: reached_account_ids, domain: banned_domains_for_misskey - friend_domains).inboxes
+  end
+
+  def reached_account_inboxes_for_friend
+    Account.where(id: reached_account_ids, domain: friend_domains).inboxes
+  end
+
+  def reached_account_ids
     # When the status is a reblog, there are no interactions with it
     # directly, we assume all interactions are with the original one
 
     if @status.reblog?
-      []
+      [reblog_of_account_id]
     elsif @status.limited_visibility?
-      Account.where(id: mentioned_account_ids).where.not(domain: banned_domains).inboxes
+      [mentioned_account_ids]
     else
-      Account.where(id: reached_account_ids).where.not(domain: banned_domains + friend_domains).inboxes
-    end
-  end
-
-  def reached_account_inboxes_for_misskey
-    if @status.reblog? || @status.limited_visibility?
-      []
-    else
-      Account.where(id: reached_account_ids, domain: banned_domains_for_misskey - friend_domains).inboxes
-    end
-  end
-
-  def reached_account_inboxes_for_friend
-    if @status.reblog? || @status.limited_visibility?
-      []
-    else
-      Account.where(id: reached_account_ids, domain: friend_domains).inboxes
-    end
-  end
-
-  def reached_account_ids
       [
         replied_to_account_id,
         reblog_of_account_id,
@@ -81,6 +72,7 @@ class StatusReachFinder
         arr.uniq!
       end
     end
+  end
 
   def replied_to_account_id
     @status.in_reply_to_account_id if distributable?

app/models/concerns/account/other_settings.rb

@@ -15,13 +15,6 @@ module Account::OtherSettings
       false
     end
 
-    def link_preview?
-      return user.setting_link_preview if local? && user.present?
-      return settings['link_preview'] if settings.present? && settings.key?('link_preview')
-
-      true
-    end
-
     def allow_quote?
       return user.setting_allow_quote if local? && user.present?
       return settings['allow_quote'] if settings.present? && settings.key?('allow_quote')
@@ -95,7 +88,6 @@ module Account::OtherSettings
         'hide_following_count' => hide_following_count?,
         'hide_followers_count' => hide_followers_count?,
         'translatable_private' => translatable_private?,
-        'link_preview' => link_preview?,
         'allow_quote' => allow_quote?,
         'emoji_reaction_policy' => Setting.enable_emoji_reaction ? emoji_reaction_policy.to_s : 'block',
       }

app/models/concerns/user/has_settings.rb

@@ -127,6 +127,10 @@ module User::HasSettings
     settings['allow_quote']
   end
 
+  def setting_reject_send_limited_to_suspects
+    settings['reject_send_limited_to_suspects']
+  end
+
   def setting_noindex
     settings['noindex']
   end
@@ -135,10 +139,6 @@ module User::HasSettings
     settings['translatable_private']
   end
 
-  def setting_link_preview
-    settings['link_preview']
-  end
-
   def setting_dtl_force_visibility
     settings['dtl_force_visibility']&.to_sym || :unchange
   end

app/models/concerns/user/omniauthable.rb

@@ -19,17 +19,18 @@ module User::Omniauthable
   end
 
   class_methods do
-    def find_for_oauth(auth, signed_in_resource = nil)
+    def find_for_omniauth(auth, signed_in_resource = nil)
       # EOLE-SSO Patch
       auth.uid = (auth.uid[0][:uid] || auth.uid[0][:user]) if auth.uid.is_a? Hashie::Array
-      identity = Identity.find_for_oauth(auth)
+      identity = Identity.find_for_omniauth(auth)
 
       # If a signed_in_resource is provided it always overrides the existing user
       # to prevent the identity being locked with accidentally created accounts.
       # Note that this may leave zombie accounts (with no associated identity) which
       # can be cleaned up at a later date.
       user   = signed_in_resource || identity.user
-      user ||= create_for_oauth(auth)
+      user ||= reattach_for_auth(auth)
+      user ||= create_for_auth(auth)
 
       if identity.user.nil?
         identity.user = user
@@ -39,19 +40,35 @@ module User::Omniauthable
       user
     end
 
-    def create_for_oauth(auth)
-      # Check if the user exists with provided email. If no email was provided,
+    private
+
+    def reattach_for_auth(auth)
+      # If allowed, check if a user exists with the provided email address,
+      # and return it if they does not have an associated identity with the
+      # current authentication provider.
+
+      # This can be used to provide a choice of alternative auth providers
+      # or provide smooth gradual transition between multiple auth providers,
+      # but this is discouraged because any insecure provider will put *all*
+      # local users at risk, regardless of which provider they registered with.
+
+      return unless ENV['ALLOW_UNSAFE_AUTH_PROVIDER_REATTACH'] == 'true'
+
+      email, email_is_verified = email_from_auth(auth)
+      return unless email_is_verified
+
+      user = User.find_by(email: email)
+      return if user.nil? || Identity.exists?(provider: auth.provider, user_id: user.id)
+
+      user
+    end
+
+    def create_for_auth(auth)
+      # Create a user for the given auth params. If no email was provided,
       # we assign a temporary email and ask the user to verify it on
       # the next step via Auth::SetupController.show
 
-      strategy          = Devise.omniauth_configs[auth.provider.to_sym].strategy
-      assume_verified   = strategy&.security&.assume_email_is_verified
-      email_is_verified = auth.info.verified || auth.info.verified_email || auth.info.email_verified || assume_verified
-      email             = auth.info.verified_email || auth.info.email
-
-      user = User.find_by(email: email) if email_is_verified
-
-      return user unless user.nil?
+      email, email_is_verified = email_from_auth(auth)
 
       user = User.new(user_params_from_auth(email, auth))
 
@@ -66,7 +83,14 @@ module User::Omniauthable
       user
     end
 
-    private
+    def email_from_auth(auth)
+      strategy          = Devise.omniauth_configs[auth.provider.to_sym].strategy
+      assume_verified   = strategy&.security&.assume_email_is_verified
+      email_is_verified = auth.info.verified || auth.info.verified_email || auth.info.email_verified || assume_verified
+      email             = auth.info.verified_email || auth.info.email
+
+      [email, email_is_verified]
+    end
 
     def user_params_from_auth(email, auth)
       {

app/models/identity.rb

@@ -17,7 +17,7 @@ class Identity < ApplicationRecord
   validates :uid, presence: true, uniqueness: { scope: :provider }
   validates :provider, presence: true
 
-  def self.find_for_oauth(auth)
+  def self.find_for_omniauth(auth)
     find_or_create_by(uid: auth.uid, provider: auth.provider)
   end
 end

app/models/instance_info.rb

@@ -17,17 +17,17 @@ class InstanceInfo < ApplicationRecord
   after_commit :reset_cache
 
   EMOJI_REACTION_AVAILABLE_SOFTWARES = %w(
-    misskey
-    calckey
-    cherrypick
-    meisskey
-    sharkey
-    firefish
-    catodon
-    renedon
-    fedibird
-    pleroma
     akkoma
+    calckey
+    catodon
+    cherrypick
+    fedibird
+    firefish
+    iceshrimp
+    meisskey
+    misskey
+    pleroma
+    sharkey
   ).freeze
 
   def self.emoji_reaction_available?(domain)

app/models/status.rb

@@ -618,7 +618,7 @@ class Status < ApplicationRecord
   end
 
   def distributable_friend?
-    public_visibility? || public_unlisted_visibility? || (unlisted_visibility? && (public_searchability? || public_unlisted_searchability?))
+    public_visibility? || public_unlisted_visibility? || login_visibility? || (unlisted_visibility? && (public_searchability? || public_unlisted_searchability?))
   end
 
   private

app/models/status_reference.rb

@@ -14,6 +14,8 @@
 #
 
 class StatusReference < ApplicationRecord
+  REFERENCES_LIMIT = 5
+
   belongs_to :status
   belongs_to :target_status, class_name: 'Status'
 

app/models/user.rb

@@ -361,6 +361,16 @@ class User < ApplicationRecord
     Doorkeeper::AccessToken.by_resource_owner(self).in_batches do |batch|
       batch.update_all(revoked_at: Time.now.utc)
       Web::PushSubscription.where(access_token_id: batch).delete_all
+
+      # Revoke each access token for the Streaming API, since `update_all``
+      # doesn't trigger ActiveRecord Callbacks:
+      # TODO: #28793 Combine into a single topic
+      payload = Oj.dump(event: :kill)
+      redis.pipelined do |pipeline|
+        batch.ids.each do |id|
+          pipeline.publish("timeline:access_token:#{id}", payload)
+        end
+      end
     end
   end
 

app/models/user_settings.rb

@@ -12,7 +12,6 @@ class UserSettings
   setting :theme, default: -> { ::Setting.theme }
   setting :noindex, default: -> { ::Setting.noindex }
   setting :translatable_private, default: false
-  setting :link_preview, default: true
   setting :bio_markdown, default: false
   setting :discoverable_local, default: false
   setting :hide_statuses_count, default: false
@@ -42,6 +41,7 @@ class UserSettings
   setting :dtl_force_subscribable, default: false
   setting :lock_follow_from_bot, default: false
   setting :allow_quote, default: true
+  setting :reject_send_limited_to_suspects, default: false
 
   setting_inverse_alias :indexable, :noindex
 

app/services/activitypub/fetch_featured_collection_service.rb

@@ -23,9 +23,9 @@ class ActivityPub::FetchFeaturedCollectionService < BaseService
 
     case collection['type']
     when 'Collection', 'CollectionPage'
-      collection['items']
+      as_array(collection['items'])
     when 'OrderedCollection', 'OrderedCollectionPage'
-      collection['orderedItems']
+      as_array(collection['orderedItems'])
     end
   end
 

app/services/activitypub/fetch_references_service.rb

@@ -6,7 +6,7 @@ class ActivityPub::FetchReferencesService < BaseService
   def call(status, collection_or_uri)
     @account = status.account
 
-    collection_items(collection_or_uri)&.map { |item| value_or_id(item) }
+    collection_items(collection_or_uri)&.take(8)&.map { |item| value_or_id(item) }
   end
 
   private
@@ -20,9 +20,9 @@ class ActivityPub::FetchReferencesService < BaseService
 
     case collection['type']
     when 'Collection', 'CollectionPage'
-      collection['items']
+      as_array(collection['items'])
     when 'OrderedCollection', 'OrderedCollectionPage'
-      collection['orderedItems']
+      as_array(collection['orderedItems'])
     end
   end
 
@@ -31,6 +31,19 @@ class ActivityPub::FetchReferencesService < BaseService
     return if unsupported_uri_scheme?(collection_or_uri)
     return if ActivityPub::TagManager.instance.local_uri?(collection_or_uri)
 
+    # NOTE: For backward compatibility reasons, Mastodon signs outgoing
+    # queries incorrectly by default.
+    #
+    # While this is relevant for all URLs with query strings, this is
+    # the only code path where this happens in practice.
+    #
+    # Therefore, retry with correct signatures if this fails.
+    begin
       fetch_resource_without_id_validation(collection_or_uri, nil, true)
+    rescue Mastodon::UnexpectedResponseError => e
+      raise unless e.response && e.response.code == 401 && Addressable::URI.parse(collection_or_uri).query.present?
+
+      fetch_resource_without_id_validation(collection_or_uri, nil, true, request_options: { with_query_string: true })
+    end
   end
 end

app/services/activitypub/fetch_remote_account_service.rb

@@ -2,7 +2,7 @@
 
 class ActivityPub::FetchRemoteAccountService < ActivityPub::FetchRemoteActorService
   # Does a WebFinger roundtrip on each call, unless `only_key` is true
-  def call(uri, id: true, prefetched_body: nil, break_on_redirect: false, only_key: false, suppress_errors: true, request_id: nil)
+  def call(uri, prefetched_body: nil, break_on_redirect: false, only_key: false, suppress_errors: true, request_id: nil)
     actor = super
     return actor if actor.nil? || actor.is_a?(Account)
 

app/services/activitypub/fetch_remote_actor_service.rb

@@ -10,15 +10,15 @@ class ActivityPub::FetchRemoteActorService < BaseService
   SUPPORTED_TYPES = %w(Application Group Organization Person Service).freeze
 
   # Does a WebFinger roundtrip on each call, unless `only_key` is true
-  def call(uri, id: true, prefetched_body: nil, break_on_redirect: false, only_key: false, suppress_errors: true, request_id: nil)
+  def call(uri, prefetched_body: nil, break_on_redirect: false, only_key: false, suppress_errors: true, request_id: nil)
     return if domain_not_allowed?(uri)
     return ActivityPub::TagManager.instance.uri_to_actor(uri) if ActivityPub::TagManager.instance.local_uri?(uri)
 
     @json = begin
       if prefetched_body.nil?
-        fetch_resource(uri, id)
+        fetch_resource(uri, true)
       else
-        body_to_json(prefetched_body, compare_id: id ? uri : nil)
+        body_to_json(prefetched_body, compare_id: uri)
       end
     rescue Oj::ParseError
       raise Error, "Error parsing JSON-LD document #{uri}"

app/services/activitypub/fetch_remote_key_service.rb

@@ -6,23 +6,10 @@ class ActivityPub::FetchRemoteKeyService < BaseService
   class Error < StandardError; end
 
   # Returns actor that owns the key
-  def call(uri, id: true, prefetched_body: nil, suppress_errors: true)
+  def call(uri, suppress_errors: true)
     raise Error, 'No key URI given' if uri.blank?
 
-    if prefetched_body.nil?
-      if id
-        @json = fetch_resource_without_id_validation(uri)
-        if actor_type?
-          @json = fetch_resource(@json['id'], true)
-        elsif uri != @json['id']
-          raise Error, "Fetched URI #{uri} has wrong id #{@json['id']}"
-        end
-      else
-        @json = fetch_resource(uri, id)
-      end
-    else
-      @json = body_to_json(prefetched_body, compare_id: id ? uri : nil)
-    end
+    @json = fetch_resource(uri, false)
 
     raise Error, "Unable to fetch key JSON at #{uri}" if @json.nil?
     raise Error, "Unsupported JSON-LD context for document #{uri}" unless supported_context?(@json)

app/services/activitypub/fetch_remote_status_service.rb

@@ -8,14 +8,14 @@ class ActivityPub::FetchRemoteStatusService < BaseService
   DISCOVERIES_PER_REQUEST = 1000
 
   # Should be called when uri has already been checked for locality
-  def call(uri, id: true, prefetched_body: nil, on_behalf_of: nil, expected_actor_uri: nil, request_id: nil)
+  def call(uri, prefetched_body: nil, on_behalf_of: nil, expected_actor_uri: nil, request_id: nil)
     return if domain_not_allowed?(uri)
 
     @request_id = request_id || "#{Time.now.utc.to_i}-status-#{uri}"
     @json = if prefetched_body.nil?
-              fetch_resource(uri, id, on_behalf_of)
+              fetch_resource(uri, true, on_behalf_of)
             else
-              body_to_json(prefetched_body, compare_id: id ? uri : nil)
+              body_to_json(prefetched_body, compare_id: uri)
             end
 
     return unless supported_context?
@@ -65,7 +65,7 @@ class ActivityPub::FetchRemoteStatusService < BaseService
 
   def account_from_uri(uri)
     actor = ActivityPub::TagManager.instance.uri_to_resource(uri, Account)
-    actor = ActivityPub::FetchRemoteAccountService.new.call(uri, id: true, request_id: @request_id) if actor.nil? || actor.possibly_stale?
+    actor = ActivityPub::FetchRemoteAccountService.new.call(uri, request_id: @request_id) if actor.nil? || actor.possibly_stale?
     actor
   end
 

app/services/activitypub/fetch_replies_service.rb

@@ -26,9 +26,9 @@ class ActivityPub::FetchRepliesService < BaseService
 
     case collection['type']
     when 'Collection', 'CollectionPage'
-      collection['items']
+      as_array(collection['items'])
     when 'OrderedCollection', 'OrderedCollectionPage'
-      collection['orderedItems']
+      as_array(collection['orderedItems'])
     end
   end
 
@@ -37,7 +37,20 @@ class ActivityPub::FetchRepliesService < BaseService
     return unless @allow_synchronous_requests
     return if non_matching_uri_hosts?(@account.uri, collection_or_uri)
 
+    # NOTE: For backward compatibility reasons, Mastodon signs outgoing
+    # queries incorrectly by default.
+    #
+    # While this is relevant for all URLs with query strings, this is
+    # the only code path where this happens in practice.
+    #
+    # Therefore, retry with correct signatures if this fails.
+    begin
       fetch_resource_without_id_validation(collection_or_uri, nil, true)
+    rescue Mastodon::UnexpectedResponseError => e
+      raise unless e.response && e.response.code == 401 && Addressable::URI.parse(collection_or_uri).query.present?
+
+      fetch_resource_without_id_validation(collection_or_uri, nil, true, request_options: { with_query_string: true })
+    end
   end
 
   def filtered_replies

app/services/activitypub/process_account_service.rb

@@ -131,8 +131,8 @@ class ActivityPub::ProcessAccountService < BaseService
   end
 
   def valid_account?
-    display_name = @json['name']
-    note = @json['summary']
+    display_name = @json['name'] || ''
+    note = @json['summary'] || ''
     !Admin::NgWord.reject?(display_name) && !Admin::NgWord.reject?(note)
   end
 
@@ -392,7 +392,7 @@ class ActivityPub::ProcessAccountService < BaseService
 
   def moved_account
     account   = ActivityPub::TagManager.instance.uri_to_resource(@json['movedTo'], Account)
-    account ||= ActivityPub::FetchRemoteAccountService.new.call(@json['movedTo'], id: true, break_on_redirect: true, request_id: @options[:request_id])
+    account ||= ActivityPub::FetchRemoteAccountService.new.call(@json['movedTo'], break_on_redirect: true, request_id: @options[:request_id])
     account
   end
 

app/services/activitypub/synchronize_followers_service.rb

@@ -59,9 +59,9 @@ class ActivityPub::SynchronizeFollowersService < BaseService
 
     case collection['type']
     when 'Collection', 'CollectionPage'
-      collection['items']
+      as_array(collection['items'])
     when 'OrderedCollection', 'OrderedCollectionPage'
-      collection['orderedItems']
+      as_array(collection['orderedItems'])
     end
   end
 

app/services/concerns/account_scope.rb

@@ -7,6 +7,8 @@ module AccountScope
       scope_local
     when :private
       scope_account_local_followers(status.account)
+    when :limited
+      scope_status_all_mentioned(status)
     else
       scope_status_mentioned(status)
     end
@@ -24,6 +26,10 @@ module AccountScope
     Account.local.where(id: status.active_mentions.select(:account_id)).reorder(nil)
   end
 
+  def scope_status_all_mentioned(status)
+    Account.local.where(id: status.mentions.select(:account_id)).reorder(nil)
+  end
+
   # TODO: not work
   def scope_list_following_account(account)
     account.lists_for_local_distribution.select(:id).reorder(nil)

app/services/fetch_link_card_service.rb

@@ -20,7 +20,7 @@ class FetchLinkCardService < BaseService
     @status       = status
     @original_url = parse_urls
 
-    return if @original_url.nil? || @status.with_preview_card? || !@status.account.link_preview?
+    return if @original_url.nil? || @status.with_preview_card?
 
     @url = @original_url.to_s
 
@@ -88,6 +88,10 @@ class FetchLinkCardService < BaseService
   end
 
   def referenced_urls
+    referenced_urls_raw.filter { |uri| ActivityPub::TagManager.instance.uri_to_resource(uri, Status, url: true).present? }
+  end
+
+  def referenced_urls_raw
     unless @status.local?
       document = Nokogiri::HTML(@status.text)
       document.search('a[href^="http://"]', 'a[href^="https://"]').each do |link|

app/services/fetch_resource_service.rb

@@ -48,7 +48,15 @@ class FetchResourceService < BaseService
       body = response.body_with_limit
       json = body_to_json(body)
 
-      [json['id'], { prefetched_body: body, id: true }] if supported_context?(json) && (equals_or_includes_any?(json['type'], ActivityPub::FetchRemoteActorService::SUPPORTED_TYPES) || expected_type?(json))
+      return unless supported_context?(json) && (equals_or_includes_any?(json['type'], ActivityPub::FetchRemoteActorService::SUPPORTED_TYPES) || expected_type?(json))
+
+      if json['id'] != @url
+        return if terminal
+
+        return process(json['id'], terminal: true)
+      end
+
+      [@url, { prefetched_body: body }]
     elsif !terminal
       link_header = response['Link'] && parse_link_header(response)
 

app/services/keys/query_service.rb

@@ -69,7 +69,7 @@ class Keys::QueryService < BaseService
 
     return if json['items'].blank?
 
-    @devices = json['items'].map do |device|
+    @devices = as_array(json['items']).map do |device|
       Device.new(device_id: device['id'], name: device['name'], identity_key: device.dig('identityKey', 'publicKeyBase64'), fingerprint_key: device.dig('fingerprintKey', 'publicKeyBase64'), claim_url: device['claim'])
     end
   rescue HTTP::Error, OpenSSL::SSL::SSLError, Mastodon::Error => e

app/services/process_mentions_service.rb

@@ -105,7 +105,10 @@ class ProcessMentionsService < BaseService
   def process_mutual!
     mentioned_account_ids = @current_mentions.map(&:account_id)
 
-    @status.account.mutuals.reorder(nil).find_each do |target_account|
+    mutuals = @status.account.mutuals
+    mutuals = mutuals.where.not(domain: InstanceInfo.where(software: 'misskey').select(:domain)).or(mutuals.where(domain: nil)) if @status.account.user&.setting_reject_send_limited_to_suspects
+
+    mutuals.reorder(nil).find_each do |target_account|
       @current_mentions << @status.mentions.new(silent: true, account: target_account) unless mentioned_account_ids.include?(target_account.id)
     end
   end

app/services/reblog_service.rb

@@ -44,11 +44,7 @@ class ReblogService < BaseService
   def create_notification(reblog)
     reblogged_status = reblog.reblog
 
-    if reblogged_status.account.local?
-      LocalNotificationWorker.perform_async(reblogged_status.account_id, reblog.id, reblog.class.name, 'reblog')
-    elsif reblogged_status.account.activitypub? && !reblogged_status.account.following?(reblog.account)
-      ActivityPub::DeliveryWorker.perform_async(build_json(reblog), reblog.account_id, reblogged_status.account.inbox_url)
-    end
+    LocalNotificationWorker.perform_async(reblogged_status.account_id, reblog.id, reblog.class.name, 'reblog') if reblogged_status.account.local?
   end
 
   def increment_statistics

app/views/settings/privacy_extra/show.html.haml

@@ -19,10 +19,16 @@
       = ff.input :translatable_private, wrapper: :with_label, kmyblue: true, label: I18n.t('simple_form.labels.defaults.setting_translatable_private')
 
     .fields-group
-      = ff.input :link_preview, wrapper: :with_label, kmyblue: true, label: I18n.t('simple_form.labels.defaults.setting_link_preview'), hint: I18n.t('simple_form.hints.defaults.setting_link_preview')
-
-    .fields-group
-      = f.input :subscription_policy, kmyblue: true, collection: %w(allow followers_only block), label_method: ->(item) { safe_join([t("simple_form.labels.subscription_policy.#{item}")]) }, as: :radio_buttons, collection_wrapper_tag: 'ul', item_wrapper_tag: 'li', wrapper: :with_floating_label, label: t('simple_form.labels.defaults.subscription_policy'), hint: t('simple_form.hints.defaults.subscription_policy')
+      = f.input :subscription_policy,
+                as: :radio_buttons,
+                collection: %w(allow followers_only block),
+                collection_wrapper_tag: 'ul',
+                hint: t('simple_form.hints.defaults.subscription_policy'),
+                item_wrapper_tag: 'li',
+                kmyblue: true,
+                label: t('simple_form.labels.defaults.subscription_policy'),
+                label_method: ->(item) { safe_join([t("simple_form.labels.subscription_policy.#{item}")]) },
+                wrapper: :with_floating_label
 
     .fields-group
       = ff.input :allow_quote, wrapper: :with_label, kmyblue: true, label: I18n.t('simple_form.labels.defaults.setting_allow_quote'), hint: false
@@ -39,5 +45,8 @@
     .fields-group
       = ff.input :reject_unlisted_subscription, kmyblue: true, as: :boolean, wrapper: :with_label, label: I18n.t('simple_form.labels.defaults.setting_reject_unlisted_subscription'), hint: I18n.t('simple_form.hints.defaults.setting_reject_unlisted_subscription')
 
+    .fields-group
+      = ff.input :reject_send_limited_to_suspects, kmyblue: true, as: :boolean, wrapper: :with_label, label: I18n.t('simple_form.labels.defaults.setting_reject_send_limited_to_suspects'), hint: I18n.t('simple_form.hints.defaults.setting_reject_send_limited_to_suspects')
+
   .actions
     = f.button :button, t('generic.save_changes'), type: :submit

app/workers/link_crawl_worker.rb

@@ -7,7 +7,7 @@ class LinkCrawlWorker
 
   def perform(status_id)
     FetchLinkCardService.new.call(Status.find(status_id))
-  rescue ActiveRecord::RecordNotFound
+  rescue ActiveRecord::RecordNotFound, ActiveRecord::RecordNotUnique
     true
   end
 end

config/initializers/doorkeeper.rb

@@ -21,9 +21,14 @@ Doorkeeper.configure do
     user unless user&.otp_required_for_login?
   end
 
-  # If you want to restrict access to the web interface for adding oauth authorized applications, you need to declare the block below.
+  # Doorkeeper provides some administrative interfaces for managing OAuth
+  # Applications, allowing creation, edit, and deletion of applications from the
+  # server. At present, these administrative routes are not integrated into
+  # Mastodon, and as such, we've disabled them by always return a 403 forbidden
+  # response for them. This does not affect the ability for users to manage
+  # their own OAuth Applications.
   admin_authenticator do
-    current_user&.admin? || redirect_to(new_user_session_url)
+    head 403
   end
 
   # Authorization Code expiration time (default 10 minutes).

config/locales/devise.en.yml

@@ -12,6 +12,7 @@ en:
       last_attempt: You have one more attempt before your account is locked.
       locked: Your account is locked.
       not_found_in_database: Invalid %{authentication_keys} or password.
+      omniauth_user_creation_failure: Error creating an account for this identity.
       pending: Your account is still under review.
       timeout: Your session expired. Please login again to continue.
       unauthenticated: You need to login or sign up before continuing.

config/locales/en.yml

@@ -2075,6 +2075,7 @@ en:
     go_to_sso_account_settings: Go to your identity provider's account settings
     invalid_otp_token: Invalid two-factor code
     otp_lost_help_html: If you lost access to both, you may get in touch with %{email}
+    rate_limited: Too many authentication attempts, try again later.
     seamless_external_login: You are logged in via an external service, so password and e-mail settings are not available.
     signed_in_as: 'Signed in as:'
   verification:

config/locales/simple_form.en.yml

@@ -278,11 +278,11 @@ en:
         setting_hide_network: Hide your social graph
         setting_hide_recent_emojis: Hide recent emojis
         setting_hide_statuses_count: Hide statuses count
-        setting_link_preview: Generate post link preview card
         setting_lock_follow_from_bot: Request approval about bot follow
         setting_public_post_to_unlisted: Convert public post to public unlisted if not using Web app
         setting_reduce_motion: Reduce motion in animations
         setting_reject_public_unlisted_subscription: Reject sending public unlisted visibility/non-public searchability posts to Misskey, Calckey
+        setting_reject_send_limited_to_suspects: Reject sending mutual posts to Misskey
         setting_reject_unlisted_subscription: Reject sending unlisted visibility/non-public searchability posts to Misskey, Calckey
         setting_send_without_domain_blocks: Send your post to all server with administrator set as rejecting-post-server for protect you [DEPRECATED]
         setting_show_application: Disclose application used to send posts

config/locales/simple_form.ja.yml

@@ -79,8 +79,8 @@ ja:
         setting_emoji_reaction_streaming_notify_impl2: 当該サーバーの独自機能に対応したアプリを利用時に、スタンプ機能を利用できます。動作確認していないため（そもそもそのようなアプリ自体を確認できていないため）正しく動かない場合があります
         setting_enable_emoji_reaction: この機能を無効にしても、他の人はあなたの投稿にスタンプをつけられます
         setting_hide_network: フォローとフォロワーの情報がプロフィールページで見られないようにします
-        setting_link_preview: プレビュー生成を停止することは、センシティブなサイトへのリンクを頻繁に投稿する人にも有効かもしれません
         setting_public_post_to_unlisted: 未対応のサードパーティアプリからもローカル公開で投稿できますが、公開投稿はWeb以外できなくなります
+        setting_reject_send_limited_to_suspects: これは「相互のみ」投稿に適用されます。サークル投稿は例外なく配送されます。一部のMisskeyサーバーが独自に限定投稿へ対応しましたが、相互のみ投稿を行うたびに相手に通知されるなど複数の問題があるため、気になる人向けの設定です
         setting_reject_unlisted_subscription: Misskeyやそのフォークは、フォローしていないアカウントの「非収載」投稿を **購読・検索** することができます。これはkmyblueの挙動と異なります。そのようなサーバーに、指定した公開範囲の投稿を「フォロワーのみ」として配送します。ただし構造上、完璧な対応は困難でたまに非収載として配信されること、ご理解ください
         setting_show_application: 投稿するのに使用したアプリが投稿の詳細ビューに表示されるようになります
         setting_single_ref_to_quote: 当サーバーがまだ対象投稿を取り込んでいない場合、引用が相手に正常に認識されない場合があります
@@ -287,7 +287,6 @@ ja:
         setting_hide_network: 繋がりを隠す
         setting_hide_recent_emojis: 絵文字ピッカーで最近使用した絵文字を隠す（絵文字デッキのみを表示する）
         setting_hide_statuses_count: 投稿数を隠す
-        setting_link_preview: リンクのプレビューを生成する
         setting_lock_follow_from_bot: botからのフォローを承認制にする
         setting_show_quote_in_home: ホーム・リスト・アンテナなどで引用を表示する
         setting_show_quote_in_public: 公開タイムライン（ローカル・連合）で引用を表示する
@@ -295,6 +294,7 @@ ja:
         setting_public_post_to_unlisted: サードパーティから公開範囲「公開」で投稿した場合、「ローカル公開」に変更する
         setting_reduce_motion: アニメーションの動きを減らす
         setting_reject_public_unlisted_subscription: Misskey系サーバーに「ローカル公開」かつ検索許可「誰でも以外」の投稿を「フォロワーのみ」に変換して配送する
+        setting_reject_send_limited_to_suspects: Misskey系サーバーに「相互のみ」投稿を配送しない
         setting_reject_unlisted_subscription: Misskey系サーバーに「非収載」かつ検索許可「誰でも以外」の投稿を「フォロワーのみ」に変換して配送する
         setting_send_without_domain_blocks: 管理人の設定した配送停止設定を拒否する (非推奨)
         setting_show_application: 送信したアプリを開示する

config/routes.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require 'sidekiq_unique_jobs/web'
+require 'sidekiq_unique_jobs/web' if ENV['ENABLE_SIDEKIQ_UNIQUE_JOBS_UI'] == true
 require 'sidekiq-scheduler/web'
 
 class RedirectWithVary < ActionDispatch::Routing::PathRedirect

docker-compose.yml

@@ -56,7 +56,7 @@ services:
 
   web:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.1
+    image: ghcr.io/mastodon/mastodon:v4.2.4
     restart: always
     env_file: .env.production
     command: bundle exec puma -C config/puma.rb
@@ -77,7 +77,7 @@ services:
 
   streaming:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.1
+    image: ghcr.io/mastodon/mastodon:v4.2.4
     restart: always
     env_file: .env.production
     command: node ./streaming
@@ -95,7 +95,7 @@ services:
 
   sidekiq:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.1
+    image: ghcr.io/mastodon/mastodon:v4.2.4
     restart: always
     env_file: .env.production
     command: bundle exec sidekiq

install/5.0/setup4.sh

@@ -6,8 +6,8 @@ Install Ruby
 EOF
 
 git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
-RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.2.2
-rbenv global 3.2.2
+RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.2.3
+rbenv global 3.2.3
 
 cat << EOF
 

install/9.0/setup4.sh

@@ -6,8 +6,8 @@ Install Ruby
 EOF
 
 git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
-RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.2.2
-rbenv global 3.2.2
+RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.2.3
+rbenv global 3.2.3
 
 cat << EOF
 

lib/mastodon/version.rb

@@ -9,7 +9,7 @@ module Mastodon
     end
 
     def kmyblue_minor
-      0
+      4
     end
 
     def kmyblue_flag
@@ -29,7 +29,7 @@ module Mastodon
     end
 
     def default_prerelease
-      'alpha.0'
+      'alpha.2'
     end
 
     def prerelease

lib/paperclip/response_with_limit_adapter.rb

@@ -16,7 +16,7 @@ module Paperclip
     private
 
     def cache_current_values
-      @original_filename = filename_from_content_disposition.presence || filename_from_path.presence || 'data'
+      @original_filename = truncated_filename
       @tempfile = copy_to_tempfile(@target)
       @content_type = ContentTypeDetector.new(@tempfile.path).detect
       @size = File.size(@tempfile)
@@ -43,6 +43,13 @@ module Paperclip
       source.response.connection.close
     end
 
+    def truncated_filename
+      filename = filename_from_content_disposition.presence || filename_from_path.presence || 'data'
+      extension = File.extname(filename)
+      basename = File.basename(filename, extension)
+      [basename[...20], extension[..4]].compact_blank.join
+    end
+
     def filename_from_content_disposition
       disposition = @target.response.headers['content-disposition']
       disposition&.match(/filename="([^"]*)"/)&.captures&.first

lib/tasks/db.rake

@@ -17,7 +17,7 @@ namespace :db do
 
   task :pre_migration_check do
     version = ActiveRecord::Base.connection.select_one("SELECT current_setting('server_version_num') AS v")['v'].to_i
-    abort 'This version of Mastodon requires PostgreSQL 9.5 or newer. Please update PostgreSQL before updating Mastodon' if version < 90_500
+    abort 'This version of Mastodon requires PostgreSQL 10.0 or newer. Please update PostgreSQL before updating Mastodon' if version < 100_000
   end
 
   Rake::Task['db:migrate'].enhance(['db:pre_migration_check'])

lib/tasks/sidekiq_unique_jobs.rake

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+namespace :sidekiq_unique_jobs do
+  task delete_all_locks: :environment do
+    digests = SidekiqUniqueJobs::Digests.new
+    digests.delete_by_pattern('*', count: digests.count)
+
+    expiring_digests = SidekiqUniqueJobs::ExpiringDigests.new
+    expiring_digests.delete_by_pattern('*', count: expiring_digests.count)
+  end
+end

spec/controllers/api/v1/streaming_controller_spec.rb

@@ -5,7 +5,7 @@ require 'rails_helper'
 describe Api::V1::StreamingController do
   around do |example|
     before = Rails.configuration.x.streaming_api_base_url
-    Rails.configuration.x.streaming_api_base_url = Rails.configuration.x.web_domain
+    Rails.configuration.x.streaming_api_base_url = "wss://#{Rails.configuration.x.web_domain}"
     example.run
     Rails.configuration.x.streaming_api_base_url = before
   end

spec/controllers/auth/sessions_controller_spec.rb

@@ -262,6 +262,26 @@ RSpec.describe Auth::SessionsController do
           end
         end
 
+        context 'when repeatedly using an invalid TOTP code before using a valid code' do
+          before do
+            stub_const('Auth::SessionsController::MAX_2FA_ATTEMPTS_PER_HOUR', 2)
+          end
+
+          it 'does not log the user in' do
+            # Travel to the beginning of an hour to avoid crossing rate-limit buckets
+            travel_to '2023-12-20T10:00:00Z'
+
+            Auth::SessionsController::MAX_2FA_ATTEMPTS_PER_HOUR.times do
+              post :create, params: { user: { otp_attempt: '1234' } }, session: { attempt_user_id: user.id, attempt_user_updated_at: user.updated_at.to_s }
+              expect(controller.current_user).to be_nil
+            end
+
+            post :create, params: { user: { otp_attempt: user.current_otp } }, session: { attempt_user_id: user.id, attempt_user_updated_at: user.updated_at.to_s }
+            expect(controller.current_user).to be_nil
+            expect(flash[:alert]).to match I18n.t('users.rate_limited')
+          end
+        end
+
         context 'when using a valid OTP' do
           before do
             post :create, params: { user: { otp_attempt: user.current_otp } }, session: { attempt_user_id: user.id, attempt_user_updated_at: user.updated_at.to_s }

spec/lib/account_statuses_filter_spec.rb

@@ -267,6 +267,80 @@ RSpec.describe AccountStatusesFilter do
       it_behaves_like 'filter params'
     end
 
+    context 'when accessed by a remote account' do
+      let(:current_account) { Fabricate(:account, uri: 'https://example.com/', domain: 'example.com') }
+      let!(:sensitive_status_with_cw) { Fabricate(:status, account: account, visibility: :public, spoiler_text: 'CW', sensitive: true) }
+      let!(:sensitive_status_with_media) do
+        Fabricate(:status, account: account, visibility: :public, sensitive: true).tap do |status|
+          Fabricate(:media_attachment, account: account, status: status)
+        end
+      end
+
+      shared_examples 'as_like_public_visibility' do
+        it 'returns private statuses, replies, and reblogs' do
+          expect(results_unique_visibilities).to match_array %w(login unlisted public_unlisted public)
+
+          expect(results_in_reply_to_ids).to_not be_empty
+
+          expect(results_reblog_of_ids).to_not be_empty
+        end
+
+        context 'when there is a direct status mentioning the non-follower' do
+          let!(:direct_status) { status_with_mention!(:direct, current_account) }
+
+          it 'returns the direct status' do
+            expect(results_ids).to include(direct_status.id)
+          end
+        end
+
+        context 'when there is a direct status mentioning other user' do
+          let!(:direct_status) { status_with_mention!(:direct) }
+
+          it 'not returns the direct status' do
+            expect(results_ids).to_not include(direct_status.id)
+          end
+        end
+
+        context 'when there is a limited status mentioning the non-follower' do
+          let!(:limited_status) { status_with_mention!(:limited, current_account) }
+
+          it 'returns the limited status' do
+            expect(results_ids).to include(limited_status.id)
+          end
+        end
+
+        context 'when there is a limited status mentioning other user' do
+          let!(:limited_status) { status_with_mention!(:limited) }
+
+          it 'not returns the limited status' do
+            expect(results_ids).to_not include(limited_status.id)
+          end
+        end
+      end
+
+      it_behaves_like 'as_like_public_visibility'
+      it_behaves_like 'filter params'
+
+      it 'returns the sensitive status' do
+        expect(results_ids).to include(sensitive_status_with_cw.id)
+        expect(results_ids).to include(sensitive_status_with_media.id)
+      end
+
+      context 'when domain-blocked reject_media' do
+        before do
+          Fabricate(:domain_block, domain: 'example.com', severity: :noop, reject_send_sensitive: true)
+        end
+
+        it_behaves_like 'as_like_public_visibility'
+        it_behaves_like 'filter params'
+
+        it 'does not return the sensitive status' do
+          expect(results_ids).to_not include(sensitive_status_with_cw.id)
+          expect(results_ids).to_not include(sensitive_status_with_media.id)
+        end
+      end
+    end
+
     private
 
     def results_unique_visibilities

spec/lib/activitypub/linked_data_signature_spec.rb

@@ -56,7 +56,7 @@ RSpec.describe ActivityPub::LinkedDataSignature do
 
         allow(ActivityPub::FetchRemoteKeyService).to receive(:new).and_return(service_stub)
 
-        allow(service_stub).to receive(:call).with('http://example.com/alice', id: false) do
+        allow(service_stub).to receive(:call).with('http://example.com/alice') do
           sender.update!(public_key: old_key)
           sender
         end
@@ -64,7 +64,7 @@ RSpec.describe ActivityPub::LinkedDataSignature do
 
       it 'fetches key and returns creator' do
         expect(subject.verify_actor!).to eq sender
-        expect(service_stub).to have_received(:call).with('http://example.com/alice', id: false).once
+        expect(service_stub).to have_received(:call).with('http://example.com/alice').once
       end
     end
 

spec/lib/activitypub/tag_manager_spec.rb

@@ -139,6 +139,14 @@ RSpec.describe ActivityPub::TagManager do
       expect(subject.cc(status)).to include(subject.uri_for(foo))
       expect(subject.cc(status)).to_not include(subject.uri_for(alice))
     end
+
+    it 'returns poster of reblogged post, if reblog' do
+      bob    = Fabricate(:account, username: 'bob', domain: 'example.com', inbox_url: 'http://example.com/bob')
+      alice  = Fabricate(:account, username: 'alice')
+      status = Fabricate(:status, visibility: :public, account: bob)
+      reblog = Fabricate(:status, visibility: :public, account: alice, reblog: status)
+      expect(subject.cc(reblog)).to include(subject.uri_for(bob))
+    end
   end
 
   describe '#cc_for_misskey' do

spec/models/account_spec.rb

@@ -397,13 +397,9 @@ RSpec.describe Account do
   describe '#public_settings_for_local' do
     subject { account.public_settings_for_local }
 
-    let(:account) { Fabricate(:user, settings: { link_preview: false, allow_quote: true, hide_statuses_count: true, emoji_reaction_policy: :followers_only }).account }
+    let(:account) { Fabricate(:user, settings: { allow_quote: true, hide_statuses_count: true, emoji_reaction_policy: :followers_only }).account }
 
     shared_examples 'some settings' do |permitted, emoji_reaction_policy|
-      it 'link_preview is disallowed' do
-        expect(subject['link_preview']).to be permitted.include?(:link_preview)
-      end
-
       it 'allow_quote is allowed' do
         expect(subject['allow_quote']).to be permitted.include?(:allow_quote)
       end
@@ -423,8 +419,14 @@ RSpec.describe Account do
 
     it_behaves_like 'some settings', %i(allow_quote hide_statuses_count), 'followers_only'
 
+    context 'when default true setting is set false' do
+      let(:account) { Fabricate(:user, settings: { allow_quote: false, hide_statuses_count: true, emoji_reaction_policy: :followers_only }).account }
+
+      it_behaves_like 'some settings', %i(hide_statuses_count), 'followers_only'
+    end
+
     context 'when remote user' do
-      let(:account) { Fabricate(:account, domain: 'example.com', uri: 'https://example.com/actor', settings: { 'link_preview' => false, 'allow_quote' => true, 'hide_statuses_count' => true, 'emoji_reaction_policy' => 'followers_only' }) }
+      let(:account) { Fabricate(:account, domain: 'example.com', uri: 'https://example.com/actor', settings: { 'allow_quote' => true, 'hide_statuses_count' => true, 'emoji_reaction_policy' => 'followers_only' }) }
 
       it_behaves_like 'some settings', %i(allow_quote hide_statuses_count), 'followers_only'
     end
@@ -432,7 +434,7 @@ RSpec.describe Account do
     context 'when remote user by server other_settings is not supported' do
       let(:account) { Fabricate(:account, domain: 'example.com', uri: 'https://example.com/actor') }
 
-      it_behaves_like 'some settings', %i(link_preview allow_quote), 'allow'
+      it_behaves_like 'some settings', %i(allow_quote), 'allow'
     end
   end
 

spec/models/identity_spec.rb

@@ -3,19 +3,19 @@
 require 'rails_helper'
 
 RSpec.describe Identity do
-  describe '.find_for_oauth' do
+  describe '.find_for_omniauth' do
     let(:auth) { Fabricate(:identity, user: Fabricate(:user)) }
 
     it 'calls .find_or_create_by' do
       allow(described_class).to receive(:find_or_create_by)
 
-      described_class.find_for_oauth(auth)
+      described_class.find_for_omniauth(auth)
 
       expect(described_class).to have_received(:find_or_create_by).with(uid: auth.uid, provider: auth.provider)
     end
 
     it 'returns an instance of Identity' do
-      expect(described_class.find_for_oauth(auth)).to be_instance_of described_class
+      expect(described_class.find_for_omniauth(auth)).to be_instance_of described_class
     end
   end
 end

spec/models/user_spec.rb

@@ -412,7 +412,10 @@ RSpec.describe User do
     let!(:access_token) { Fabricate(:access_token, resource_owner_id: user.id) }
     let!(:web_push_subscription) { Fabricate(:web_push_subscription, access_token: access_token) }
 
+    let(:redis_pipeline_stub) { instance_double(Redis::Namespace, publish: nil) }
+
     before do
+      allow(redis).to receive(:pipelined).and_yield(redis_pipeline_stub)
       user.reset_password!
     end
 
@@ -429,6 +432,10 @@ RSpec.describe User do
       expect(Doorkeeper::AccessToken.active_for(user).count).to eq 0
     end
 
+    it 'revokes streaming access for all access tokens' do
+      expect(redis_pipeline_stub).to have_received(:publish).with("timeline:access_token:#{access_token.id}", Oj.dump(event: :kill)).once
+    end
+
     it 'removes push subscriptions' do
       expect(Web::PushSubscription.where(user: user).or(Web::PushSubscription.where(access_token: access_token)).count).to eq 0
       expect { web_push_subscription.reload }.to raise_error(ActiveRecord::RecordNotFound)

spec/requests/disabled_oauth_endpoints_spec.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+require 'rails_helper'
+
+describe 'Disabled OAuth routes' do
+  # These routes are disabled via the doorkeeper configuration for
+  # `admin_authenticator`, as these routes should only be accessible by server
+  # administrators. For now, these routes are not properly designed and
+  # integrated into Mastodon, so we're disabling them completely
+  describe 'GET /oauth/applications' do
+    it 'returns 403 forbidden' do
+      get oauth_applications_path
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'POST /oauth/applications' do
+    it 'returns 403 forbidden' do
+      post oauth_applications_path
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'GET /oauth/applications/new' do
+    it 'returns 403 forbidden' do
+      get new_oauth_application_path
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'GET /oauth/applications/:id' do
+    let(:application) { Fabricate(:application, scopes: 'read') }
+
+    it 'returns 403 forbidden' do
+      get oauth_application_path(application)
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'PATCH /oauth/applications/:id' do
+    let(:application) { Fabricate(:application, scopes: 'read') }
+
+    it 'returns 403 forbidden' do
+      patch oauth_application_path(application)
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'PUT /oauth/applications/:id' do
+    let(:application) { Fabricate(:application, scopes: 'read') }
+
+    it 'returns 403 forbidden' do
+      put oauth_application_path(application)
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'DELETE /oauth/applications/:id' do
+    let(:application) { Fabricate(:application, scopes: 'read') }
+
+    it 'returns 403 forbidden' do
+      delete oauth_application_path(application)
+
+      expect(response).to have_http_status(403)
+    end
+  end
+
+  describe 'GET /oauth/applications/:id/edit' do
+    let(:application) { Fabricate(:application, scopes: 'read') }
+
+    it 'returns 403 forbidden' do
+      get edit_oauth_application_path(application)
+
+      expect(response).to have_http_status(403)
+    end
+  end
+end

spec/requests/omniauth_callbacks_spec.rb

@@ -39,6 +39,13 @@ describe 'OmniAuth callbacks' do
           Fabricate(:user, email: 'user@host.example')
         end
 
+        context 'when ALLOW_UNSAFE_AUTH_PROVIDER_REATTACH is set to true' do
+          around do |example|
+            ClimateControl.modify ALLOW_UNSAFE_AUTH_PROVIDER_REATTACH: 'true' do
+              example.run
+            end
+          end
+
           it 'matches the existing user, creates an identity, and redirects to root path' do
             expect { subject }
               .to not_change(User, :count)
@@ -52,6 +59,18 @@ describe 'OmniAuth callbacks' do
           end
         end
 
+        context 'when ALLOW_UNSAFE_AUTH_PROVIDER_REATTACH is not set to true' do
+          it 'does not match the existing user or create an identity, and redirects to login page' do
+            expect { subject }
+              .to not_change(User, :count)
+              .and not_change(Identity, :count)
+              .and not_change(LoginActivity, :count)
+
+            expect(response).to redirect_to(new_user_session_url)
+          end
+        end
+      end
+
       context 'with a matching user and a matching identity' do
         before do
           user = Fabricate(:user, email: 'user@host.example')
@@ -96,7 +115,7 @@ describe 'OmniAuth callbacks' do
 
     context 'when a user cannot be built' do
       before do
-        allow(User).to receive(:find_for_oauth).and_return(User.new)
+        allow(User).to receive(:find_for_omniauth).and_return(User.new)
       end
 
       it 'redirects to the new user signup page' do

spec/services/activitypub/fetch_featured_collection_service_spec.rb

@@ -31,7 +31,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
     }
   end
 
-  let(:status_json_pinned_unknown_unreachable) do
+  let(:status_json_pinned_unknown_reachable) do
     {
       '@context': 'https://www.w3.org/ns/activitystreams',
       type: 'Note',
@@ -75,7 +75,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
       stub_request(:get, 'https://example.com/account/pinned/known').to_return(status: 200, body: Oj.dump(status_json_pinned_known))
       stub_request(:get, 'https://example.com/account/pinned/unknown-inlined').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_inlined))
       stub_request(:get, 'https://example.com/account/pinned/unknown-unreachable').to_return(status: 404)
-      stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_unreachable))
+      stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable))
       stub_request(:get, 'https://example.com/account/collections/featured').to_return(status: 200, body: Oj.dump(featured_with_null))
 
       subject.call(actor, note: true, hashtag: false)
@@ -115,6 +115,21 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
       end
 
       it_behaves_like 'sets pinned posts'
+
+      context 'when there is a single item, with the array compacted away' do
+        let(:items) { 'https://example.com/account/pinned/unknown-reachable' }
+
+        before do
+          stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable))
+          subject.call(actor, note: true, hashtag: false)
+        end
+
+        it 'sets expected posts as pinned posts' do
+          expect(actor.pinned_statuses.pluck(:uri)).to contain_exactly(
+            'https://example.com/account/pinned/unknown-reachable'
+          )
+        end
+      end
     end
 
     context 'when the endpoint is a paginated Collection' do
@@ -136,6 +151,21 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
       end
 
       it_behaves_like 'sets pinned posts'
+
+      context 'when there is a single item, with the array compacted away' do
+        let(:items) { 'https://example.com/account/pinned/unknown-reachable' }
+
+        before do
+          stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable))
+          subject.call(actor, note: true, hashtag: false)
+        end
+
+        it 'sets expected posts as pinned posts' do
+          expect(actor.pinned_statuses.pluck(:uri)).to contain_exactly(
+            'https://example.com/account/pinned/unknown-reachable'
+          )
+        end
+      end
     end
   end
 end

spec/services/activitypub/fetch_references_service_spec.rb

@@ -0,0 +1,128 @@
+# frozen_string_literal: true
+
+require 'rails_helper'
+
+RSpec.describe ActivityPub::FetchReferencesService, type: :service do
+  subject { described_class.new.call(status, payload) }
+
+  let(:actor)          { Fabricate(:account, domain: 'example.com', uri: 'http://example.com/account') }
+  let(:status)         { Fabricate(:status, account: actor) }
+  let(:collection_uri) { 'http://example.com/references/1' }
+
+  let(:items) do
+    [
+      'http://example.com/self-references-1',
+      'http://example.com/self-references-2',
+      'http://example.com/self-references-3',
+      'http://other.com/other-references-1',
+      'http://other.com/other-references-2',
+      'http://other.com/other-references-3',
+      'http://example.com/self-references-4',
+      'http://example.com/self-references-5',
+      'http://example.com/self-references-6',
+      'http://example.com/self-references-7',
+      'http://example.com/self-references-8',
+    ]
+  end
+
+  let(:payload) do
+    {
+      '@context': 'https://www.w3.org/ns/activitystreams',
+      type: 'Collection',
+      id: collection_uri,
+      items: items,
+    }.with_indifferent_access
+  end
+
+  describe '#call' do
+    context 'when the payload is a Collection with inlined replies' do
+      context 'when there is a single reference, with the array compacted away' do
+        let(:items) { 'http://example.com/self-references-1' }
+
+        it 'a item is returned' do
+          expect(subject).to eq ['http://example.com/self-references-1']
+        end
+      end
+
+      context 'when passing the collection itself' do
+        it 'first 8 items are returned' do
+          expect(subject).to eq items.take(8)
+        end
+      end
+
+      context 'when passing the URL to the collection' do
+        subject { described_class.new.call(status, collection_uri) }
+
+        before do
+          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+        end
+
+        it 'first 8 items are returned' do
+          expect(subject).to eq items.take(8)
+        end
+      end
+    end
+
+    context 'when the payload is an OrderedCollection with inlined references' do
+      let(:payload) do
+        {
+          '@context': 'https://www.w3.org/ns/activitystreams',
+          type: 'OrderedCollection',
+          id: collection_uri,
+          orderedItems: items,
+        }.with_indifferent_access
+      end
+
+      context 'when passing the collection itself' do
+        it 'first 8 items are returned' do
+          expect(subject).to eq items.take(8)
+        end
+      end
+
+      context 'when passing the URL to the collection' do
+        subject { described_class.new.call(status, collection_uri) }
+
+        before do
+          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+        end
+
+        it 'first 8 items are returned' do
+          expect(subject).to eq items.take(8)
+        end
+      end
+    end
+
+    context 'when the payload is a paginated Collection with inlined references' do
+      let(:payload) do
+        {
+          '@context': 'https://www.w3.org/ns/activitystreams',
+          type: 'Collection',
+          id: collection_uri,
+          first: {
+            type: 'CollectionPage',
+            partOf: collection_uri,
+            items: items,
+          },
+        }.with_indifferent_access
+      end
+
+      context 'when passing the collection itself' do
+        it 'first 8 items are returned' do
+          expect(subject).to eq items.take(8)
+        end
+      end
+
+      context 'when passing the URL to the collection' do
+        subject { described_class.new.call(status, collection_uri) }
+
+        before do
+          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+        end
+
+        it 'first 8 items are returned' do
+          expect(subject).to eq items.take(8)
+        end
+      end
+    end
+  end
+end

spec/services/activitypub/fetch_remote_account_service_spec.rb

@@ -18,7 +18,7 @@ RSpec.describe ActivityPub::FetchRemoteAccountService, type: :service do
   end
 
   describe '#call' do
-    let(:account) { subject.call('https://example.com/alice', id: true) }
+    let(:account) { subject.call('https://example.com/alice') }
 
     shared_examples 'sets profile data' do
       it 'returns an account' do

spec/services/activitypub/fetch_remote_actor_service_spec.rb

@@ -18,7 +18,7 @@ RSpec.describe ActivityPub::FetchRemoteActorService, type: :service do
   end
 
   describe '#call' do
-    let(:account) { subject.call('https://example.com/alice', id: true) }
+    let(:account) { subject.call('https://example.com/alice') }
 
     shared_examples 'sets profile data' do
       it 'returns an account' do

spec/services/activitypub/fetch_remote_key_service_spec.rb

@@ -56,7 +56,7 @@ RSpec.describe ActivityPub::FetchRemoteKeyService, type: :service do
   end
 
   describe '#call' do
-    let(:account) { subject.call(public_key_id, id: false) }
+    let(:account) { subject.call(public_key_id) }
 
     context 'when the key is a sub-object from the actor' do
       before do

spec/services/activitypub/fetch_replies_service_spec.rb

@@ -34,6 +34,18 @@ RSpec.describe ActivityPub::FetchRepliesService, type: :service do
 
   describe '#call' do
     context 'when the payload is a Collection with inlined replies' do
+      context 'when there is a single reply, with the array compacted away' do
+        let(:items) { 'http://example.com/self-reply-1' }
+
+        it 'queues the expected worker' do
+          allow(FetchReplyWorker).to receive(:push_bulk)
+
+          subject.call(status, payload)
+
+          expect(FetchReplyWorker).to have_received(:push_bulk).with(['http://example.com/self-reply-1'])
+        end
+      end
+
       context 'when passing the collection itself' do
         it 'spawns workers for up to 5 replies on the same server' do
           allow(FetchReplyWorker).to receive(:push_bulk)

spec/services/activitypub/process_account_service_spec.rb

@@ -282,6 +282,33 @@ RSpec.describe ActivityPub::ProcessAccountService, type: :service do
     end
   end
 
+  context 'when account is using note contains ng words' do
+    subject { described_class.new.call(account.username, account.domain, payload) }
+
+    let!(:account) { Fabricate(:account, username: 'alice', domain: 'example.com') }
+
+    let(:payload) do
+      {
+        id: 'https://foo.test',
+        type: 'Actor',
+        inbox: 'https://foo.test/inbox',
+        name: 'Ohagi',
+      }.with_indifferent_access
+    end
+
+    it 'creates account when ng word is not set' do
+      Setting.ng_words = ['Amazon']
+      subject
+      expect(account.reload.display_name).to eq 'Ohagi'
+    end
+
+    it 'does not create account when ng word is set' do
+      Setting.ng_words = ['Ohagi']
+      subject
+      expect(account.reload.display_name).to_not eq 'Ohagi'
+    end
+  end
+
   context 'when account is not suspended' do
     subject { described_class.new.call(account.username, account.domain, payload) }
 

spec/services/fetch_link_card_service_spec.rb

@@ -7,6 +7,7 @@ RSpec.describe FetchLinkCardService, type: :service do
 
   let(:html) { '<!doctype html><title>Hello world</title>' }
   let(:oembed_cache) { nil }
+  let(:custom_before) { false }
 
   before do
     stub_request(:get, 'http://example.com/html').to_return(headers: { 'Content-Type' => 'text/html' }, body: html)
@@ -30,7 +31,7 @@ RSpec.describe FetchLinkCardService, type: :service do
 
     Rails.cache.write('oembed_endpoint:example.com', oembed_cache) if oembed_cache
 
-    subject.call(status)
+    subject.call(status) unless custom_before
   end
 
   context 'with a local status' do
@@ -236,32 +237,53 @@ RSpec.describe FetchLinkCardService, type: :service do
       end
     end
 
-    context 'with URL of reference' do
-      let(:status) { Fabricate(:status, text: 'RT http://example.com/html') }
-
-      it 'creates preview card' do
-        expect(status.preview_card).to be_nil
-      end
-    end
-
-    context 'with URL of reference and normal page' do
+    context 'with URI of reference and normal page' do
       let(:status) { Fabricate(:status, text: 'RT http://example.com/text http://example.com/html') }
+      let(:custom_before) { true }
+
+      before { Fabricate(:status, uri: 'http://example.com/text') }
 
       it 'creates preview card' do
+        subject.call(status)
         expect(status.preview_card).to_not be_nil
         expect(status.preview_card.url).to eq 'http://example.com/html'
         expect(status.preview_card.title).to eq 'Hello world'
       end
     end
 
-    context 'with URL but author is not allow preview card' do
-      let(:account) { Fabricate(:user, settings: { link_preview: false }).account }
-      let(:status) { Fabricate(:status, text: 'http://example.com/html', account: account) }
+    context 'with URI of reference' do
+      let(:status) { Fabricate(:status, text: 'RT http://example.com/text') }
+      let(:custom_before) { true }
 
-      it 'not create preview card' do
+      before { Fabricate(:status, uri: 'http://example.com/text') }
+
+      it 'does not create preview card' do
+        subject.call(status)
         expect(status.preview_card).to be_nil
       end
     end
+
+    context 'with URL of reference' do
+      let(:status) { Fabricate(:status, text: 'RT http://example.com/text') }
+      let(:custom_before) { true }
+
+      before { Fabricate(:status, uri: 'http://example.com/text/activity', url: 'http://example.com/text') }
+
+      it 'does not create preview card' do
+        subject.call(status)
+        expect(status.preview_card).to be_nil
+      end
+    end
+
+    context 'with reference normal URL' do
+      let(:status) { Fabricate(:status, text: 'RT http://example.com/html') }
+
+      it 'creates preview card' do
+        expect(status.preview_card).to_not be_nil
+        expect(status.preview_card.url).to eq 'http://example.com/html'
+        expect(status.preview_card.title).to eq 'Hello world'
+      end
+    end
   end
 
   context 'with a remote status' do
@@ -282,14 +304,6 @@ RSpec.describe FetchLinkCardService, type: :service do
     it 'ignores URLs to hashtags' do
       expect(a_request(:get, 'https://quitter.se/tag/wannacry')).to_not have_been_made
     end
-
-    context 'with URL but author is not allow preview card' do
-      let(:account) { Fabricate(:account, domain: 'example.com', settings: { link_preview: false }) }
-
-      it 'not create link preview' do
-        expect(status.preview_card).to be_nil
-      end
-    end
   end
 
   context 'with a remote status of reference' do
@@ -298,8 +312,12 @@ RSpec.describe FetchLinkCardService, type: :service do
       RT <a href="http://example.com/html" target="_blank" rel="noopener noreferrer" title="http://example.com/html">Hello</a>&nbsp;
       TEXT
     end
+    let(:custom_before) { true }
+
+    before { Fabricate(:status, uri: 'http://example.com/html') }
 
     it 'creates preview card' do
+      subject.call(status)
       expect(status.preview_card).to be_nil
     end
   end
@@ -311,8 +329,12 @@ RSpec.describe FetchLinkCardService, type: :service do
       <a href="http://example.com/html_sub" target="_blank" rel="noopener noreferrer" title="http://example.com/html_sub">Hello</a>&nbsp;
       TEXT
     end
+    let(:custom_before) { true }
+
+    before { Fabricate(:status, uri: 'http://example.com/html') }
 
     it 'creates preview card' do
+      subject.call(status)
       expect(status.preview_card).to_not be_nil
       expect(status.preview_card.url).to eq 'http://example.com/html_sub'
       expect(status.preview_card.title).to eq 'Hello world'

spec/services/fetch_resource_service_spec.rb

@@ -57,7 +57,7 @@ RSpec.describe FetchResourceService, type: :service do
 
       let(:json) do
         {
-          id: 1,
+          id: 'http://example.com/foo',
           '@context': ActivityPub::TagManager::CONTEXT,
           type: 'Note',
         }.to_json
@@ -83,27 +83,27 @@ RSpec.describe FetchResourceService, type: :service do
         let(:content_type) { 'application/activity+json; charset=utf-8' }
         let(:body) { json }
 
-        it { is_expected.to eq [1, { prefetched_body: body, id: true }] }
+        it { is_expected.to eq ['http://example.com/foo', { prefetched_body: body }] }
       end
 
       context 'when content type is ld+json with profile' do
         let(:content_type) { 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"' }
         let(:body) { json }
 
-        it { is_expected.to eq [1, { prefetched_body: body, id: true }] }
+        it { is_expected.to eq ['http://example.com/foo', { prefetched_body: body }] }
       end
 
       context 'when link header is present' do
         let(:headers) { { 'Link' => '<http://example.com/foo>; rel="alternate"; type="application/activity+json"' } }
 
-        it { is_expected.to eq [1, { prefetched_body: json, id: true }] }
+        it { is_expected.to eq ['http://example.com/foo', { prefetched_body: json }] }
       end
 
       context 'when content type is text/html' do
         let(:content_type) { 'text/html' }
         let(:body) { '<html><head><link rel="alternate" href="http://example.com/foo" type="application/activity+json"/></head></html>' }
 
-        it { is_expected.to eq [1, { prefetched_body: json, id: true }] }
+        it { is_expected.to eq ['http://example.com/foo', { prefetched_body: json }] }
       end
     end
   end

spec/services/post_status_service_spec.rb

@@ -192,21 +192,50 @@ RSpec.describe PostStatusService, type: :service do
     expect(mention_service).to have_received(:call).with(status, limited_type: '', circle: nil, save_records: false)
   end
 
-  it 'mutual visibility' do
-    account = Fabricate(:account)
-    mutual_account = Fabricate(:account)
-    other_account = Fabricate(:account)
-    text = 'This is an English text.'
+  context 'with mutual visibility' do
+    let(:sender) { Fabricate(:user).account }
+    let(:io_account) { Fabricate(:account, domain: 'misskey.io', uri: 'https://misskey.io/actor', inbox_url: 'https://misskey.io/inbox') }
+    let(:local_account) { Fabricate(:account) }
+    let(:remote_account) { Fabricate(:account, domain: 'example.com', uri: 'https://example.com/actor', inbox_url: 'https://example.com/inbox') }
+    let(:follower) { Fabricate(:account) }
+    let(:followee) { Fabricate(:account) }
 
-    mutual_account.follow!(account)
-    account.follow!(mutual_account)
-    other_account.follow!(account)
-    status = subject.call(account, text: text, visibility: 'mutual')
+    before do
+      stub_request(:post, 'https://misskey.io/inbox').to_return(status: 200)
+      stub_request(:post, 'https://example.com/inbox').to_return(status: 200)
+      Fabricate(:instance_info, domain: 'misskey.io', software: 'misskey')
+      io_account.follow!(sender)
+      local_account.follow!(sender)
+      remote_account.follow!(sender)
+      follower.follow!(sender)
+      sender.follow!(io_account)
+      sender.follow!(local_account)
+      sender.follow!(remote_account)
+      sender.follow!(followee)
+    end
+
+    it 'visibility is set' do
+      status = subject.call(sender, text: 'text', visibility: 'mutual')
 
       expect(status.visibility).to eq 'limited'
       expect(status.limited_scope).to eq 'mutual'
-    expect(status.mentioned_accounts.count).to eq 1
-    expect(status.mentioned_accounts.first.id).to eq mutual_account.id
+    end
+
+    it 'sent to mutuals' do
+      status = subject.call(sender, text: 'text', visibility: 'mutual')
+
+      expect(status.mentioned_accounts.count).to eq 3
+      expect(status.mentioned_accounts.pluck(:id)).to contain_exactly(io_account.id, local_account.id, remote_account.id)
+    end
+
+    it 'sent to mutuals without misskey.io users' do
+      sender.user.update!(settings: { reject_send_limited_to_suspects: true })
+
+      status = subject.call(sender, text: 'text', visibility: 'mutual')
+
+      expect(status.mentioned_accounts.count).to eq 2
+      expect(status.mentioned_accounts.pluck(:id)).to contain_exactly(local_account.id, remote_account.id)
+    end
   end
 
   it 'limited visibility and direct searchability' do

spec/services/reblog_service_spec.rb

@@ -86,9 +86,5 @@ RSpec.describe ReblogService, type: :service do
     it 'distributes to followers' do
       expect(ActivityPub::DistributionWorker).to have_received(:perform_async)
     end
-
-    it 'sends an announce activity to the author' do
-      expect(a_request(:post, bob.inbox_url)).to have_been_made.once
-    end
   end
 end

spec/services/remove_status_service_spec.rb

@@ -160,4 +160,22 @@ RSpec.describe RemoveStatusService, type: :service do
              )).to have_been_made.once
     end
   end
+
+  context 'when removed status is a reblog of a non-follower' do
+    let!(:original_status) { Fabricate(:status, account: bill, text: 'Hello ThisIsASecret', visibility: :public) }
+    let!(:status) { ReblogService.new.call(alice, original_status) }
+
+    it 'sends Undo activity to followers' do
+      subject.call(status)
+      expect(a_request(:post, bill.shared_inbox_url).with(
+               body: hash_including({
+                 'type' => 'Undo',
+                 'object' => hash_including({
+                   'type' => 'Announce',
+                   'object' => ActivityPub::TagManager.instance.uri_for(original_status),
+                 }),
+               })
+             )).to have_been_made.once
+    end
+  end
 end

spec/services/resolve_url_service_spec.rb

@@ -139,6 +139,7 @@ describe ResolveURLService, type: :service do
         stub_request(:get, url).to_return(status: 302, headers: { 'Location' => status_url })
         body = ActiveModelSerializers::SerializableResource.new(status, serializer: ActivityPub::NoteSerializer, adapter: ActivityPub::Adapter).to_json
         stub_request(:get, status_url).to_return(body: body, headers: { 'Content-Type' => 'application/activity+json' })
+        stub_request(:get, uri).to_return(body: body, headers: { 'Content-Type' => 'application/activity+json' })
       end
 
       it 'returns status by url' do