Scala algorithm: Remove duplicates from a sorted list (Sliding)

Published

Algorithm goal

In Streaming data, data may come in duplicated; it could be due to various factors such as duplicated data from sources and idempotency for redundancy; for consumption though we may need to deduplicate the data for at-most-once processing. Some deduplicators retain state of long-gone elements (in which case .distinct will suffice, but have a memory cost), but in this case we are looking at only consecutive duplicate elements.

This is an alternative approach to RemoveDuplicatesFromSortedListStateMachine

Test cases in Scala

This algorithm comes with test cases.
To see the free test cases for all our algos, as well as run them in our TDD IDE, Register (free).

Algorithm in Scala

23 lines of Scala (compatible versions 2.13 & 3.0), showing how concise Scala can be!

Get the full algorithm Scala algorithms logo, maze part, which looks quirky!

or

'Unlimited Scala Algorithms' gives you access to all the 77 published & 15 upcoming Scala Algorithms (92 total)! Plenty to master your Scala algos skills!

Upon purchase, you will be able to Register an account to access all the algorithms on multiple devices.

Stripe logo

Explanation

The approach here is a little different to the State Machine approach:

The idea is that by running a sliding window, we can check pairs of elements for their equality, and if they are not equal, emit the newly found one. (this is © from www.scala-algorithms.com)

Full explanation is available for subscribers Scala algorithms logo, maze part, which looks quirky

Scala concepts & Hints

  1. Collect

    'collect' allows you to use Pattern Matching, to filter and map items.

    assert("Hello World".collect {
  case character if Character.isUpperCase(character) => character.toLower
} == "hw")

  2. Option Type

    The 'Option' type is used to describe a computation that either has a result or does not. In Scala, you can 'chain' Option processing, combine with lists and other data structures. For example, you can also turn a pattern-match into a function that return an Option, and vice-versa!

    assert(Option(1).flatMap(x => Option(x + 2)) == Option(3))

assert(Option(1).flatMap(x => None) == None)

  3. Pattern Matching

    Pattern matching in Scala lets you quickly identify what you are looking for in a data, and also extract it.

    assert("Hello World".collect {
  case character if Character.isUpperCase(character) => character.toLower
} == "hw")

  4. Sliding / Sliding Window

    Get fixed-length sliding sub-sequences (sliding windows) from another sequence

Scala Algorithms: The most comprehensive library of algorithms in standard pure-functional Scala

Think in Scala & master the highest paid programming language in the US

Scala is used at many places, such as AirBnB, Apple, Bank of America, BBC, Barclays, Capital One, Citibank, Coursera, eBay, JP Morgan, LinkedIn, Morgan Stanley, Netflix, Singapore Exchange, Twitter.

Study our 92 Scala Algorithms: 6 fully free, 77 published & 15 upcoming

Fully unit-tested, with explanations and relevant concepts; new algorithms published about once a week.

  1. Compute the length of longest valid parentheses
  2. Check a binary tree is balanced
  3. Make a queue using stacks (Lists in Scala)
  4. Find height of binary tree
  5. Single-elimination tournament tree
  6. Quick Sort sorting algorithm in pure immutable Scala
  7. Find minimum missing positive number in a sequence
  8. Least-recently used cache (LRU)
  9. Count pairs of a given expected sum
  10. Compute a Roman numeral for an Integer, and vice-versa
  11. Compute keypad possibilities
  12. Matching parentheses algorithm with foldLeft and a state machine
  13. Traverse a tree Breadth-First, immutably
  14. Read a matrix as a spiral
  15. Remove duplicates from a sorted list (state machine)
  16. Merge Sort: stack-safe, tail-recursive, in pure immutable Scala, N-way
  17. Longest increasing sub-sequence length
  18. Reverse first n elements of a queue
  19. Binary search a generic Array
  20. Merge Sort: in pure immutable Scala
  21. Make a queue using Maps
  22. Is an Array a permutation?
  23. Count number of contiguous countries by colors
  24. Add numbers without using addition (plus sign)
  25. Tic Tac Toe MinMax solve
  26. Run-length encoding (RLE) Encoder
  27. Print Alphabet Diamond
  28. Balanced parentheses algorithm with tail-call recursion optimisation
  29. Reverse a String's words efficiently
  30. Count number of changes (manipulations) needed to make an anagram with foldLeft and a MultiSet
  31. Count passing cars
  32. Establish execution order from dependencies
  33. Counting inversions of a sequence (array) using a Merge Sort
  34. Longest common prefix of strings
  35. Check if an array is a palindrome
  36. Check a directed graph has a routing between two nodes (depth-first search)
  37. Compute nth row of Pascal's triangle
  38. Run-length encoding (RLE) Decoder
  39. Check if a number is a palindrome
  40. In a range of numbers, count the numbers divisible by a specific integer
  41. Find the index of a substring ('indexOf')
  42. Reshape a matrix
  43. Compute modulo of an exponent without exponentiation
  44. Closest pair of coordinates in a 2D plane
  45. Find the contiguous slice with the minimum average
  46. Compute maximum sum of subarray (Kadane's algorithm)
  47. Pure-functional double linked list
  48. Binary search in a rotated sorted array
  49. Check if a directed graph has cycles
  50. Rotate Array right in pure-functional Scala - using an unusual immutable efficient approach
  51. Check a binary tree is a search tree
  52. Length of the longest common substring
  53. Sliding Window Rate Limiter
  54. Tic Tac Toe board check
  55. Find an unpaired number in an array
  56. Check if a String is a palindrome
  57. Count binary gap size of a number using tail recursion
  58. Remove duplicates from a sorted list (Sliding)
  59. Monitor success rate of a process that may fail
  60. Find sub-array with the maximum sum
  61. Find the minimum absolute difference of two partitions
  62. Find maximum potential profit from an array of stock price
  63. Fibonacci in purely functional immutable Scala
  64. Fizz Buzz in purely functional immutable Scala
  65. Find combinations adding up to N (non-unique)
  66. Make a binary search tree (Red-Black tree)
  67. Remove duplicates from an unsorted List
  68. Mars Rover
  69. Find combinations adding up to N (unique)
  70. Find indices of tuples that sum to a target (Two Sum)
  71. Count factors/divisors of an integer
  72. Compute single-digit sum of digits
  73. Traverse a tree Depth-First
  74. Reverse bits of an integer
  75. Find k closest elements to a value in a sorted Array
  76. QuickSelect Selection Algorithm (kth smallest item/order statistic)
  77. Rotate a matrix by 90 degrees clockwise

Explore the 21 most useful Scala concepts

To save you going through various tutorials, we cherry-picked the most useful Scala concepts in a consistent form.

  1. Class Inside Class
  2. Class Inside Def
  3. Collect
  4. Def Inside Def
  5. Drop, Take, dropRight, takeRight
  6. foldLeft and foldRight
  7. For-comprehension
  8. Lazy List
  9. Option Type
  10. Ordering
  11. Partial Function
  12. Pattern Matching
  13. Range
  14. scanLeft and scanRight
  15. Sliding / Sliding Window
  16. Stack Safety
  17. State machine
  18. Tail Recursion
  19. Type Class
  20. View
  21. Zip

Register now (free)